API Reference¶
This page contains the API documentation for all Python modules in the codebase (excluding init.py files).
aiperf.cli¶
Main CLI entry point for the AIPerf system.
analyze(user_config, service_config=None)
¶
Sweep through one or more parameters.
Source code in aiperf/cli.py
42 43 44 45 46 47 48 49 50 51 | |
create_template(template_filename=CLIDefaults.TEMPLATE_FILENAME)
¶
Create a template configuration file.
Source code in aiperf/cli.py
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
profile(user_config, service_config=None)
¶
Run the Profile subcommand.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
user_config
|
UserConfig
|
User configuration for the benchmark |
required |
service_config
|
ServiceConfig | None
|
Service configuration options |
None
|
Source code in aiperf/cli.py
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | |
validate_config(user_config=None, service_config=None)
¶
Validate the configuration file.
Source code in aiperf/cli.py
73 74 75 76 77 78 79 80 81 82 | |
aiperf.cli_runner¶
run_system_controller(user_config, service_config)
¶
Run the system controller with the given configuration.
Source code in aiperf/cli_runner.py
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | |
warn_command_not_implemented(command)
¶
Warn the user that the subcommand is not implemented.
Source code in aiperf/cli_runner.py
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | |
aiperf.clients.client_interfaces¶
InferenceClientFactory
¶
Bases: FactoryMixin[EndpointType, InferenceClientProtocol]
Factory for registering and creating InferenceClientProtocol instances based on the specified endpoint type.
see: :class:FactoryMixin for more details.
Source code in aiperf/clients/client_interfaces.py
56 57 58 59 | |
InferenceClientProtocol
¶
Bases: Protocol, Generic[RequestInputT]
Protocol for an inference server client.
This protocol defines the methods that must be implemented by any inference server client implementation that is compatible with the AIPerf framework.
Source code in aiperf/clients/client_interfaces.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | |
__init__(model_endpoint)
¶
Create a new inference server client based on the provided configuration.
Source code in aiperf/clients/client_interfaces.py
26 27 28 | |
close()
async
¶
Close the client.
Source code in aiperf/clients/client_interfaces.py
51 52 53 | |
initialize()
async
¶
Initialize the inference server client in an asynchronous context.
Source code in aiperf/clients/client_interfaces.py
30 31 32 | |
send_request(model_endpoint, payload)
async
¶
Send a request to the inference server.
This method is used to send a request to the inference server.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_endpoint
|
ModelEndpointInfo
|
The endpoint to send the request to. |
required |
payload
|
RequestInputT
|
The payload to send to the inference server. |
required |
Returns: The raw response from the inference server.
Source code in aiperf/clients/client_interfaces.py
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
RequestConverterFactory
¶
Bases: FactoryMixin[EndpointType, RequestConverterProtocol]
Factory for registering and creating RequestConverterProtocol instances based on the specified request payload type.
see: :class:FactoryMixin for more details.
Source code in aiperf/clients/client_interfaces.py
78 79 80 81 | |
RequestConverterProtocol
¶
Bases: Protocol, Generic[RequestOutputT]
Protocol for a request converter that converts a raw request to a formatted request for the inference server.
Source code in aiperf/clients/client_interfaces.py
67 68 69 70 71 72 73 74 75 | |
format_payload(model_endpoint, turn)
async
¶
Format the turn for the inference server.
Source code in aiperf/clients/client_interfaces.py
71 72 73 74 75 | |
ResponseExtractorFactory
¶
Bases: FactoryMixin[EndpointType, ResponseExtractorProtocol]
Factory for registering and creating ResponseExtractorProtocol instances based on the specified response extractor type.
see: :class:FactoryMixin for more details.
Source code in aiperf/clients/client_interfaces.py
101 102 103 104 | |
ResponseExtractorProtocol
¶
Bases: Protocol
Protocol for a response extractor that extracts the response data from a raw inference server response and converts it to a list of ResponseData objects.
Source code in aiperf/clients/client_interfaces.py
89 90 91 92 93 94 95 96 97 98 | |
extract_response_data(record, tokenizer)
async
¶
Extract the response data from a raw inference server response and convert it to a list of ResponseData objects.
Source code in aiperf/clients/client_interfaces.py
94 95 96 97 98 | |
aiperf.clients.http.aiohttp_client¶
AioHttpClientMixin
¶
A high-performance HTTP client for communicating with HTTP based REST APIs using aiohttp.
This class is optimized for maximum performance and accurate timing measurements, making it ideal for benchmarking scenarios.
Source code in aiperf/clients/http/aiohttp_client.py
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |
close()
async
¶
Close the client.
Source code in aiperf/clients/http/aiohttp_client.py
50 51 52 53 54 | |
post_request(url, payload, headers, **kwargs)
async
¶
Send a streaming or non-streaming POST request to the specified URL with the given payload and headers.
If the response is an SSE stream, the response will be parsed into a list of SSE messages. Otherwise, the response will be parsed into a TextResponse object.
Source code in aiperf/clients/http/aiohttp_client.py
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | |
AioHttpSSEStreamReader
¶
A helper class for reading an SSE stream from an aiohttp.ClientResponse object.
This class is optimized for maximum performance and accurate timing measurements, making it ideal for benchmarking scenarios.
Source code in aiperf/clients/http/aiohttp_client.py
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | |
__aiter__()
async
¶
Iterate over the SSE stream in a performant manner and return a tuple of the raw SSE message, the perf_counter_ns of the first byte, and the perf_counter_ns of the last byte. This provides the most accurate timing information possible without any delays due to the nature of the aiohttp library. The first byte is read immediately to capture the timestamp of the first byte, and the last byte is read after the rest of the chunk is read to capture the timestamp of the last byte.
Returns:
| Type | Description |
|---|---|
AsyncIterator[tuple[str, int]]
|
An async iterator of tuples of the raw SSE message, and the perf_counter_ns of the first byte |
Source code in aiperf/clients/http/aiohttp_client.py
157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | |
read_complete_stream()
async
¶
Read the complete SSE stream in a performant manner and return a list of SSE messages that contain the most accurate timestamp data possible.
Returns:
| Type | Description |
|---|---|
list[SSEMessage]
|
A list of SSE messages. |
Source code in aiperf/clients/http/aiohttp_client.py
141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 | |
create_tcp_connector(**kwargs)
¶
Create a new connector with the given configuration.
Source code in aiperf/clients/http/aiohttp_client.py
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 | |
parse_sse_message(raw_message, perf_ns)
¶
Parse a raw SSE message into an SSEMessage object.
Parsing logic based on official HTML SSE Living Standard: https://html.spec.whatwg.org/multipage/server-sent-events.html#parsing-an-event-stream
Source code in aiperf/clients/http/aiohttp_client.py
195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 | |
aiperf.clients.http.defaults¶
AioHttpDefaults
dataclass
¶
Default values for aiohttp.ClientSession.
Source code in aiperf/clients/http/defaults.py
62 63 64 65 66 67 68 69 70 71 72 73 74 | |
SocketDefaults
dataclass
¶
Default values for socket options.
Source code in aiperf/clients/http/defaults.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | |
apply_to_socket(sock)
classmethod
¶
Apply the default socket options to the given socket.
Source code in aiperf/clients/http/defaults.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | |
aiperf.clients.model_endpoint_info¶
Model endpoint information.
This module contains the pydantic models that encapsulate the information needed to send requests to an inference server, primarily around the model, endpoint, and additional request payload information.
EndpointInfo
¶
Bases: AIPerfBaseModel
Information about an endpoint.
Source code in aiperf/clients/model_endpoint_info.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | |
from_user_config(user_config)
classmethod
¶
Create an HttpEndpointInfo from a UserConfig.
Source code in aiperf/clients/model_endpoint_info.py
106 107 108 109 110 111 112 113 114 115 116 117 118 | |
ModelEndpointInfo
¶
Bases: AIPerfBaseModel
Information about a model endpoint.
Source code in aiperf/clients/model_endpoint_info.py
121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | |
primary_model
property
¶
Get the primary model.
primary_model_name
property
¶
Get the primary model name.
url
property
¶
Get the full URL for the endpoint.
from_user_config(user_config)
classmethod
¶
Create a ModelEndpointInfo from a UserConfig.
Source code in aiperf/clients/model_endpoint_info.py
133 134 135 136 137 138 139 | |
ModelInfo
¶
Bases: AIPerfBaseModel
Information about a model.
Source code in aiperf/clients/model_endpoint_info.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | |
ModelListInfo
¶
Bases: AIPerfBaseModel
Information about a list of models.
Source code in aiperf/clients/model_endpoint_info.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | |
from_user_config(user_config)
classmethod
¶
Create a ModelListInfo from a UserConfig.
Source code in aiperf/clients/model_endpoint_info.py
52 53 54 55 56 57 58 | |
aiperf.clients.openai.openai_aiohttp¶
OpenAIClientAioHttp
¶
Bases: AioHttpClientMixin, ABC
Inference client for OpenAI based requests using aiohttp.
Source code in aiperf/clients/openai/openai_aiohttp.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
get_headers(model_endpoint)
¶
Get the headers for the given endpoint.
Source code in aiperf/clients/openai/openai_aiohttp.py
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | |
get_url(model_endpoint)
¶
Get the URL for the given endpoint.
Source code in aiperf/clients/openai/openai_aiohttp.py
55 56 57 58 59 60 | |
send_request(model_endpoint, payload)
async
¶
Send OpenAI request using aiohttp.
Source code in aiperf/clients/openai/openai_aiohttp.py
62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
aiperf.clients.openai.openai_chat¶
OpenAIChatCompletionRequestConverter
¶
Bases: RequestConverterProtocol[dict[str, Any]]
Request converter for OpenAI chat completion requests.
Source code in aiperf/clients/openai/openai_chat.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | |
format_payload(model_endpoint, turn)
async
¶
Format payload for a chat completion request.
Source code in aiperf/clients/openai/openai_chat.py
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | |
aiperf.clients.openai.openai_completions¶
OpenAICompletionRequestConverter
¶
Bases: RequestConverterProtocol[dict[str, Any]]
Request converter for OpenAI completion requests.
Source code in aiperf/clients/openai/openai_completions.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
format_payload(model_endpoint, turn)
async
¶
Format payload for a completion request.
Source code in aiperf/clients/openai/openai_completions.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
aiperf.clients.openai.openai_responses¶
OpenAIResponsesRequestConverter
¶
Bases: RequestConverterProtocol[dict[str, Any]]
Request converter for OpenAI Responses requests.
Source code in aiperf/clients/openai/openai_responses.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
format_payload(model_endpoint, turn)
async
¶
Format payload for a responses request.
Source code in aiperf/clients/openai/openai_responses.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
aiperf.common.aiperf_logger¶
AIPerfLogger
¶
Logger for AIPerf messages with lazy evaluation support for f-strings.
This logger supports lazy evaluation of f-strings through lambdas to avoid expensive string formatting operations when the log level is not enabled.
It also extends the standard logging module with additional log levels
- TRACE (TRACE < DEBUG)
- NOTICE (INFO < NOTICE < WARNING)
- SUCCESS (WARNING < SUCCESS < ERROR)
Usage
logger = AIPerfLogger("my_logger") logger.debug(lambda: f"Processing {item} with {count} items") logger.info("Simple string message") logger.notice("Notice message") logger.success("Benchmark completed successfully")
Need to pass local variables to the lambda to avoid them going out of scope¶
logger.debug(lambda i=i: f"Binding loop variable: {i}") logger.exception(f"Direct f-string usage: {e}")
Source code in aiperf/common/aiperf_logger.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 | |
critical(msg, *args, **kwargs)
¶
Log a critical message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
202 203 204 205 | |
debug(msg, *args, **kwargs)
¶
Log a debug message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
167 168 169 170 | |
error(msg, *args, **kwargs)
¶
Log an error message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
192 193 194 195 | |
exception(msg, *args, **kwargs)
¶
Log an exception message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
197 198 199 200 | |
find_caller(stack_info=False, stacklevel=1)
¶
NOTE: This is a modified version of the findCaller method in the logging module, in order to allow us to add custom ignored files.
Find the stack frame of the caller so that we can note the source file name, line number and function name.
Source code in aiperf/common/aiperf_logger.py
114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 | |
get_level_number(level)
classmethod
¶
Get the numeric level for the given level.
Source code in aiperf/common/aiperf_logger.py
106 107 108 109 110 111 112 | |
info(msg, *args, **kwargs)
¶
Log an info message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
172 173 174 175 | |
is_valid_level(level)
classmethod
¶
Check if the given level is a valid level.
Source code in aiperf/common/aiperf_logger.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | |
log(level, msg, *args, **kwargs)
¶
Log a message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
157 158 159 160 | |
notice(msg, *args, **kwargs)
¶
Log a notice message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
177 178 179 180 | |
success(msg, *args, **kwargs)
¶
Log a success message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
187 188 189 190 | |
trace(msg, *args, **kwargs)
¶
Log a trace message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
162 163 164 165 | |
warning(msg, *args, **kwargs)
¶
Log a warning message with support for lazy evaluation using lambdas.
Source code in aiperf/common/aiperf_logger.py
182 183 184 185 | |
aiperf.common.bootstrap¶
bootstrap_and_run_service(service_class, service_config=None, user_config=None, service_id=None, log_queue=None, **kwargs)
¶
Bootstrap the service and run it.
This function will load the service configuration, create an instance of the service, and run it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_class
|
type[BaseService]
|
The python class of the service to run. This should be a subclass of BaseService. This should be a type and not an instance. |
required |
service_config
|
ServiceConfig | None
|
The service configuration to use. If not provided, the service configuration will be loaded from the environment variables. |
None
|
user_config
|
UserConfig | None
|
The user configuration to use. If not provided, the user configuration will be loaded from the environment variables. |
None
|
log_queue
|
Queue | None
|
Optional multiprocessing queue for child process logging. If provided, the child process logging will be set up. |
None
|
kwargs
|
Additional keyword arguments to pass to the service constructor. |
{}
|
Source code in aiperf/common/bootstrap.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | |
aiperf.common.comms.base¶
BaseCommunication
¶
Bases: ABC
Base class for specifying the base communication layer for AIPerf components.
Source code in aiperf/common/comms/base.py
223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 | |
is_initialized
abstractmethod
property
¶
Check if communication channels are initialized.
Returns:
| Type | Description |
|---|---|
bool
|
True if communication channels are initialized, False otherwise |
stop_requested
abstractmethod
property
¶
Check if the communication channels are being shutdown.
Returns:
| Type | Description |
|---|---|
bool
|
True if the communication channels are being shutdown, False otherwise |
create_client(client_type, address, bind=False, socket_ops=None)
abstractmethod
¶
Create a communication client for a given client type and address.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client_type
|
CommunicationClientType
|
The type of client to create. |
required |
address
|
CommunicationClientAddressType | str
|
The type of address to use when looking up in the communication config, or the address itself. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
False
|
socket_ops
|
dict | None
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/base.py
263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 | |
get_address(address_type)
abstractmethod
¶
Get the address for a given address type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
address_type
|
CommunicationClientAddressType | str
|
The type of address to get the address for, or the address itself. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The address for the given address type, or the address itself if it is a string. |
Source code in aiperf/common/comms/base.py
252 253 254 255 256 257 258 259 260 261 | |
initialize()
abstractmethod
async
¶
Initialize communication channels.
Source code in aiperf/common/comms/base.py
226 227 228 | |
shutdown()
abstractmethod
async
¶
Gracefully shutdown communication channels.
Source code in aiperf/common/comms/base.py
248 249 250 | |
CommunicationClientFactory
¶
Bases: FactoryMixin[CommunicationClientType, CommunicationClientProtocol]
Factory for registering and creating BaseCommunicationClient instances based on the specified client type.
Example:
# Register a new client type
@CommunicationClientFactory.register(ClientType.PUB)
class ZMQPubClient(BaseZMQClient):
pass
# Create a new client instance
client = CommunicationClientFactory.create_instance(
ClientType.PUB,
address=ClientAddressType.SERVICE_XSUB_FRONTEND,
bind=False,
)
Source code in aiperf/common/comms/base.py
166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 | |
CommunicationClientProtocol
¶
Bases: Protocol
Base interface for specifying the base communication client for AIPerf components.
Source code in aiperf/common/comms/base.py
26 27 28 29 30 31 32 33 34 35 36 | |
initialize()
async
¶
Initialize communication channels.
Source code in aiperf/common/comms/base.py
30 31 32 | |
shutdown()
async
¶
Shutdown communication channels.
Source code in aiperf/common/comms/base.py
34 35 36 | |
CommunicationClientProtocolFactory
¶
Bases: FactoryMixin[CommunicationClientType, CommunicationClientProtocol]
Factory for registering CommunicationClientProtocol interfaces for dynamic client creation.
Source code in aiperf/common/comms/base.py
39 40 41 42 | |
CommunicationFactory
¶
Bases: FactoryMixin[CommunicationBackend, BaseCommunication]
Factory for registering and creating BaseCommunication instances based on the specified communication backend.
See :class:FactoryMixin for more details.
Source code in aiperf/common/comms/base.py
300 301 302 303 | |
PubClientProtocol
¶
Bases: CommunicationClientProtocol
Interface for publish clients.
Source code in aiperf/common/comms/base.py
152 153 154 155 156 157 158 | |
publish(message)
async
¶
Publish a message. The message will be routed automatically based on the message type.
Source code in aiperf/common/comms/base.py
156 157 158 | |
PullClientProtocol
¶
Bases: CommunicationClientProtocol
Interface for pull clients.
Source code in aiperf/common/comms/base.py
60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | |
register_pull_callback(message_type, callback, max_concurrency=None)
async
¶
Register a callback for a pull client. The callback will be called when a message is received for the given message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message_type
|
MessageType
|
The message type to register the callback for |
required |
callback
|
Callable[[MessageT], Coroutine[Any, Any, None]]
|
The callback to register |
required |
max_concurrency
|
int | None
|
The maximum number of concurrent requests to allow. |
None
|
Source code in aiperf/common/comms/base.py
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | |
PushClientProtocol
¶
Bases: CommunicationClientProtocol
Interface for push clients.
Source code in aiperf/common/comms/base.py
45 46 47 48 49 50 51 52 53 54 55 56 57 | |
push(message)
async
¶
Push data to a target. The message will be routed automatically based on the message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message_type
|
MessageType to push to |
required | |
message
|
Message
|
Message to be pushed |
required |
Source code in aiperf/common/comms/base.py
49 50 51 52 53 54 55 56 57 | |
ReplyClientProtocol
¶
Bases: CommunicationClientProtocol
Interface for reply clients.
Source code in aiperf/common/comms/base.py
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | |
register_request_handler(service_id, message_type, handler)
¶
Register a request handler for a message type. The handler will be called when a request is received for the given message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id
|
str
|
The service ID to register the handler for |
required |
message_type
|
MessageType
|
The message type to register the handler for |
required |
handler
|
Callable[[MessageT], Coroutine[Any, Any, MessageOutputT | None]]
|
The handler to register |
required |
Source code in aiperf/common/comms/base.py
121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | |
RequestClientProtocol
¶
Bases: CommunicationClientProtocol
Interface for request clients.
Source code in aiperf/common/comms/base.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | |
request(message, timeout=DEFAULT_COMMS_REQUEST_TIMEOUT)
async
¶
Send a request and wait for a response. The message will be routed automatically based on the message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
MessageT
|
Message to send (will be routed based on the message type) |
required |
timeout
|
float
|
Timeout in seconds for the request. |
DEFAULT_COMMS_REQUEST_TIMEOUT
|
Returns:
| Type | Description |
|---|---|
MessageOutputT
|
Response message if successful |
Source code in aiperf/common/comms/base.py
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | |
request_async(message, callback)
async
¶
Send a request and be notified when the response is received. The message will be routed automatically based on the message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
MessageT
|
Message to send (will be routed based on the message type) |
required |
callback
|
Callable[[MessageOutputT], Coroutine[Any, Any, None]]
|
Callback to be called when the response is received |
required |
Source code in aiperf/common/comms/base.py
102 103 104 105 106 107 108 109 110 111 112 113 114 | |
SubClientProtocol
¶
Bases: CommunicationClientProtocol
Interface for subscribe clients.
Source code in aiperf/common/comms/base.py
138 139 140 141 142 143 144 145 146 147 148 149 | |
subscribe(message_type, callback)
async
¶
Subscribe to a specific message type. The callback will be called when a message is received for the given message type.
Source code in aiperf/common/comms/base.py
142 143 144 145 146 147 148 149 | |
aiperf.common.comms.zmq.dealer_request_client¶
ZMQDealerRequestClient
¶
Bases: BaseZMQClient, AsyncTaskManagerMixin
ZMQ DEALER socket client for asynchronous request-response communication.
The DEALER socket connects to ROUTER sockets and can send requests asynchronously, receiving responses through callbacks or awaitable futures.
ASCII Diagram: ┌──────────────┐ ┌──────────────┐ │ DEALER │───── Request ─────>│ ROUTER │ │ (Client) │ │ (Service) │ │ │<─── Response ──────│ │ └──────────────┘ └──────────────┘
Usage Pattern: - DEALER Clients send requests to ROUTER Services - Responses are routed back to the originating DEALER
DEALER/ROUTER is a Many-to-One communication pattern. If you need Many-to-Many,
use a ZMQ Proxy as well. see :class:ZMQDealerRouterProxy for more details.
Source code in aiperf/common/comms/zmq/dealer_request_client.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | |
__init__(context, address, bind, socket_ops=None)
¶
Initialize the ZMQ Dealer (Req) client class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/dealer_request_client.py
44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | |
request(message, timeout=DEFAULT_COMMS_REQUEST_TIMEOUT)
async
¶
Send a request and wait for a response up to timeout seconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
Message
|
The request message to send. |
required |
timeout
|
float
|
Maximum time to wait for a response in seconds. |
DEFAULT_COMMS_REQUEST_TIMEOUT
|
Returns:
| Name | Type | Description |
|---|---|---|
Message |
Message
|
The response message received. |
Raises:
| Type | Description |
|---|---|
CommunicationError
|
if the request fails, or |
TimeoutError
|
if the response is not received in time. |
Source code in aiperf/common/comms/zmq/dealer_request_client.py
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 | |
request_async(message, callback)
async
¶
Send a request and be notified when the response is received.
Source code in aiperf/common/comms/zmq/dealer_request_client.py
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | |
aiperf.common.comms.zmq.pub_client¶
ZMQPubClient
¶
Bases: BaseZMQClient
The PUB socket broadcasts messages to all connected SUB sockets that have subscribed to the message topic/type.
ASCII Diagram: ┌──────────────┐ ┌──────────────┐ │ PUB │───>│ │ │ (Publisher) │ │ │ └──────────────┘ │ SUB │ ┌──────────────┐ │ (Subscriber) │ │ PUB │───>│ │ │ (Publisher) │ │ │ └──────────────┘ └──────────────┘ OR ┌──────────────┐ ┌──────────────┐ │ │───>│ SUB │ │ │ │ (Subscriber) │ │ PUB │ └──────────────┘ │ (Publisher) │ ┌──────────────┐ │ │───>│ SUB │ │ │ │ (Subscriber) │ └──────────────┘ └──────────────┘
Usage Pattern: - Single PUB socket broadcasts messages to all subscribers (One-to-Many) OR - Multiple PUB sockets broadcast messages to a single SUB socket (Many-to-One)
- SUB sockets filter messages by topic/message_type
- Fire-and-forget messaging (no acknowledgments)
PUB/SUB is a One-to-Many communication pattern. If you need Many-to-Many,
use a ZMQ Proxy as well. see :class:ZMQXPubXSubProxy for more details.
Source code in aiperf/common/comms/zmq/pub_client.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | |
__init__(context, address, bind, socket_ops=None)
¶
Initialize the ZMQ Publisher client class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/pub_client.py
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | |
publish(message)
async
¶
Publish a message. The topic will be set automatically based on the message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
Message
|
Message to publish (must be a Message object) |
required |
Source code in aiperf/common/comms/zmq/pub_client.py
70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | |
aiperf.common.comms.zmq.pull_client¶
ZMQPullClient
¶
Bases: BaseZMQClient, AsyncTaskManagerMixin
ZMQ PULL socket client for receiving work from PUSH sockets.
The PULL socket receives messages from PUSH sockets in a pipeline pattern, distributing work fairly among multiple PULL workers.
ASCII Diagram: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ PUSH │ │ PULL │ │ PULL │ │ (Producer) │ │ (Worker 1) │ │ (Worker 2) │ │ │ └─────────────┘ └─────────────┘ │ Tasks: │ ▲ ▲ │ - Task A │─────────────┘ │ │ - Task B │───────────────────────────────────┘ │ - Task C │─────────────┐ │ - Task D │ ▼ └─────────────┘ ┌─────────────┐ │ PULL │ │ (Worker N) │ └─────────────┘
Usage Pattern: - PULL receives work from multiple PUSH producers - Work is fairly distributed among PULL workers - Pipeline pattern for distributed processing - Each message is delivered to exactly one PULL socket
PULL/PUSH is a One-to-Many communication pattern. If you need Many-to-Many,
use a ZMQ Proxy as well. see :class:ZMQPushPullProxy for more details.
Source code in aiperf/common/comms/zmq/pull_client.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | |
__init__(context, address, bind, socket_ops=None, max_concurrency=None)
¶
Initialize the ZMQ Puller class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
max_concurrency
|
int
|
The maximum number of concurrent requests to allow. |
None
|
Source code in aiperf/common/comms/zmq/pull_client.py
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | |
register_pull_callback(message_type, callback, max_concurrency=None)
async
¶
Register a ZMQ Pull data callback for a given message type.
Note that only one callback can be registered for a given message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message_type
|
MessageType
|
The message type to register the callback for. |
required |
callback
|
Callable[[Message], Coroutine[Any, Any, None]]
|
The function to call when data is received. |
required |
max_concurrency
|
int | None
|
The maximum number of concurrent requests to allow. |
None
|
Raises: CommunicationError: If the client is not initialized
Source code in aiperf/common/comms/zmq/pull_client.py
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | |
aiperf.common.comms.zmq.push_client¶
MAX_PUSH_RETRIES = 2
module-attribute
¶
Maximum number of retries for pushing a message.
RETRY_DELAY_INTERVAL_SEC = 0.1
module-attribute
¶
The interval to wait before retrying to push a message.
ZMQPushClient
¶
Bases: BaseZMQClient, AsyncTaskManagerMixin
ZMQ PUSH socket client for sending work to PULL sockets.
The PUSH socket sends messages to PULL sockets in a pipeline pattern, distributing work fairly among available PULL workers.
ASCII Diagram: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ PUSH │ │ PULL │ │ PULL │ │ (Producer) │ │ (Worker 1) │ │ (Worker 2) │ │ │ └─────────────┘ └─────────────┘ │ Tasks: │ ▲ ▲ │ - Task A │─────────────┘ │ │ - Task B │───────────────────────────────────┘ │ - Task C │─────────────┐ │ - Task D │ ▼ └─────────────┘ ┌─────────────┐ │ PULL │ │ (Worker 3) │ └─────────────┘
Usage Pattern: - Round-robin distribution of work tasks (One-to-Many) - Each message delivered to exactly one worker - Pipeline pattern for distributed processing - Automatic load balancing across available workers
PUSH/PULL is a One-to-Many communication pattern. If you need Many-to-Many,
use a ZMQ Proxy as well. see :class:ZMQPushPullProxy for more details.
Source code in aiperf/common/comms/zmq/push_client.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | |
__init__(context, address, bind, socket_ops=None)
¶
Initialize the ZMQ Push client class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/push_client.py
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
push(message)
async
¶
Push data to a target. The message will be routed automatically based on the message type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
Message
|
Message to be sent must be a Message object |
required |
Source code in aiperf/common/comms/zmq/push_client.py
103 104 105 106 107 108 109 110 111 112 | |
aiperf.common.comms.zmq.router_reply_client¶
ZMQRouterReplyClient
¶
Bases: BaseZMQClient, AsyncTaskManagerMixin
ZMQ ROUTER socket client for handling requests from DEALER clients.
The ROUTER socket receives requests from DEALER clients and sends responses back to the originating DEALER client using routing envelopes.
ASCII Diagram: ┌──────────────┐ ┌──────────────┐ │ DEALER │───── Request ─────>│ │ │ (Client) │<──── Response ─────│ │ └──────────────┘ │ │ ┌──────────────┐ │ ROUTER │ │ DEALER │───── Request ─────>│ (Service) │ │ (Client) │<──── Response ─────│ │ └──────────────┘ │ │ ┌──────────────┐ │ │ │ DEALER │───── Request ─────>│ │ │ (Client) │<──── Response ─────│ │ └──────────────┘ └──────────────┘
Usage Pattern: - ROUTER handles requests from multiple DEALER clients - Maintains routing envelopes to send responses back - Many-to-one request handling pattern - Supports concurrent request processing
ROUTER/DEALER is a Many-to-One communication pattern. If you need Many-to-Many,
use a ZMQ Proxy as well. see :class:ZMQDealerRouterProxy for more details.
Source code in aiperf/common/comms/zmq/router_reply_client.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 | |
__init__(context, address, bind, socket_ops=None)
¶
Initialize the ZMQ Router (Rep) client class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/router_reply_client.py
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | |
register_request_handler(service_id, message_type, handler)
¶
Register a request handler. Anytime a request is received that matches the message type, the handler will be called. The handler should return a response message. If the handler returns None, the request will be ignored.
Note that there is a limit of 1 to 1 mapping between message type and handler.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id
|
str
|
The service ID to register the handler for |
required |
message_type
|
MessageType
|
The message type to register the handler for |
required |
handler
|
Callable[[Message], Coroutine[Any, Any, Message | None]]
|
The handler to register |
required |
Source code in aiperf/common/comms/zmq/router_reply_client.py
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | |
aiperf.common.comms.zmq.sub_client¶
ZMQSubClient
¶
Bases: BaseZMQClient, AsyncTaskManagerMixin
ZMQ SUB socket client for subscribing to messages from PUB sockets. One-to-Many or Many-to-One communication pattern.
ASCII Diagram: ┌──────────────┐ ┌──────────────┐ │ PUB │───>│ │ │ (Publisher) │ │ │ └──────────────┘ │ SUB │ ┌──────────────┐ │ (Subscriber) │ │ PUB │───>│ │ │ (Publisher) │ │ │ └──────────────┘ └──────────────┘ OR ┌──────────────┐ ┌──────────────┐ │ │───>│ SUB │ │ │ │ (Subscriber) │ │ PUB │ └──────────────┘ │ (Publisher) │ ┌──────────────┐ │ │───>│ SUB │ │ │ │ (Subscriber) │ └──────────────┘ └──────────────┘
Usage Pattern: - Single SUB socket subscribes to multiple PUB publishers (One-to-Many) OR - Multiple SUB sockets subscribe to a single PUB publisher (Many-to-One)
- Subscribes to specific message topics/types
- Receives all messages matching subscriptions
SUB/PUB is a One-to-Many communication pattern. If you need Many-to-Many,
use a ZMQ Proxy as well. see :class:ZMQXPubXSubProxy for more details.
Source code in aiperf/common/comms/zmq/sub_client.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 | |
__init__(context, address, bind, socket_ops=None)
¶
Initialize the ZMQ Subscriber class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/sub_client.py
58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | |
subscribe(message_type, callback)
async
¶
Subscribe to a message_type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message_type
|
MessageType
|
MessageType to subscribe to |
required |
callback
|
Callable[[Message], Any]
|
Function to call when a message is received (receives Message object) |
required |
Source code in aiperf/common/comms/zmq/sub_client.py
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | |
subscribe_all(message_callback_map)
async
¶
Subscribe to all message_types in the map.
Source code in aiperf/common/comms/zmq/sub_client.py
82 83 84 85 86 87 88 89 90 91 92 | |
aiperf.common.comms.zmq.zmq_base_client¶
BaseZMQClient
¶
Bases: AIPerfTaskMixin, AIPerfLoggerMixin
Base class for all ZMQ clients. It can be used as-is to create a new ZMQ client, or it can be subclassed to create specific ZMQ client functionality.
It inherits from the :class:AIPerfTaskMixin, allowing derived
classes to implement specific hooks.
Source code in aiperf/common/comms/zmq/zmq_base_client.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 | |
is_initialized
property
¶
Check if the client is initialized.
socket
property
¶
Get the zmq socket for the client.
Raises:
| Type | Description |
|---|---|
CommunicationError
|
If the client is not initialized |
socket_type_name
property
¶
Get the name of the socket type.
stop_requested
property
¶
Check if the client has been requested to stop.
__init__(context, socket_type, address, bind, socket_ops=None, client_id=None)
¶
Initialize the ZMQ Base class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Context
|
The ZMQ context. |
required |
address
|
str
|
The address to bind or connect to. |
required |
bind
|
bool
|
Whether to BIND or CONNECT the socket. |
required |
socket_type
|
SocketType
|
The type of ZMQ socket (eg. PUB, SUB, ROUTER, DEALER, etc.). |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/zmq_base_client.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | |
initialize()
async
¶
Initialize the communication.
This method will: - Create the zmq socket - Bind or connect the socket to the address - Set the socket options - Run the AIPerfHook.ON_INIT hooks
Source code in aiperf/common/comms/zmq/zmq_base_client.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 | |
shutdown()
async
¶
Shutdown the communication.
This method will: - Close the zmq socket - Run the AIPerfHook.ON_CLEANUP hooks
Source code in aiperf/common/comms/zmq/zmq_base_client.py
167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 | |
aiperf.common.comms.zmq.zmq_comms¶
BaseZMQCommunication
¶
Bases: BaseCommunication, AIPerfLoggerMixin, ABC
ZeroMQ-based implementation of the Communication interface.
Uses ZeroMQ for publish/subscribe and request/reply patterns to facilitate communication between AIPerf components.
Source code in aiperf/common/comms/zmq/zmq_comms.py
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | |
is_initialized
property
¶
Check if communication channels are initialized.
stop_requested
property
¶
Check if the communication channels are being shutdown.
create_client(client_type, address, bind=False, socket_ops=None)
¶
Create a communication client for a given client type and address.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client_type
|
CommunicationClientType
|
The type of client to create. |
required |
address
|
CommunicationClientAddressType | str
|
The type of address to use when looking up in the communication config, or the address itself. |
required |
bind
|
bool
|
Whether to bind or connect the socket. |
False
|
socket_ops
|
dict | None
|
Additional socket options to set. |
None
|
Source code in aiperf/common/comms/zmq/zmq_comms.py
126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | |
get_address(address_type)
¶
Get the actual address based on the address type from the config.
Source code in aiperf/common/comms/zmq/zmq_comms.py
61 62 63 64 65 | |
initialize()
async
¶
Initialize communication channels.
Source code in aiperf/common/comms/zmq/zmq_comms.py
67 68 69 70 71 72 73 74 75 76 77 78 | |
shutdown()
async
¶
Gracefully shutdown communication channels.
This method will wait for all clients to shutdown before shutting down the context.
Returns:
| Type | Description |
|---|---|
None
|
True if shutdown was successful, False otherwise |
Source code in aiperf/common/comms/zmq/zmq_comms.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | |
ZMQIPCCommunication
¶
Bases: BaseZMQCommunication
ZeroMQ-based implementation of the Communication interface using IPC transport.
Source code in aiperf/common/comms/zmq/zmq_comms.py
166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 | |
__init__(config=None)
¶
Initialize ZMQ IPC communication.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
ZMQIPCConfig | None
|
ZMQIPCConfig object with configuration parameters |
None
|
Source code in aiperf/common/comms/zmq/zmq_comms.py
170 171 172 173 174 175 176 177 178 | |
initialize()
async
¶
Initialize communication channels.
This method will create the IPC socket directory if needed.
Source code in aiperf/common/comms/zmq/zmq_comms.py
180 181 182 183 184 185 | |
shutdown()
async
¶
Gracefully shutdown communication channels.
This method will wait for all clients to shutdown before shutting down the context.
Source code in aiperf/common/comms/zmq/zmq_comms.py
187 188 189 190 191 192 193 194 | |
ZMQTCPCommunication
¶
Bases: BaseZMQCommunication
ZeroMQ-based implementation of the Communication interface using TCP transport.
Source code in aiperf/common/comms/zmq/zmq_comms.py
153 154 155 156 157 158 159 160 161 162 163 | |
__init__(config=None)
¶
Initialize ZMQ TCP communication.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
ZMQTCPConfig | None
|
ZMQTCPTransportConfig object with configuration parameters |
None
|
Source code in aiperf/common/comms/zmq/zmq_comms.py
157 158 159 160 161 162 163 | |
aiperf.common.comms.zmq.zmq_defaults¶
ZMQSocketDefaults
¶
Default values for ZMQ sockets.
Source code in aiperf/common/comms/zmq/zmq_defaults.py
5 6 7 8 9 10 11 12 13 14 15 16 | |
aiperf.common.comms.zmq.zmq_proxy_base¶
BaseZMQProxy
¶
Bases: AIPerfLoggerMixin, ABC
A Base ZMQ Proxy class.
- Frontend and backend sockets forward messages bidirectionally
- Frontend and Backend sockets both BIND
- Multiple clients CONNECT to
frontend_address - Multiple services CONNECT to
backend_address - Control: Optional REP socket for proxy commands (start/stop/pause) - not implemented yet
- Monitoring: Optional PUB socket that broadcasts copies of all forwarded messages - not implemented yet
- Proxy runs in separate thread to avoid blocking main event loop
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | |
__init__(frontend_socket_class, backend_socket_class, context, zmq_proxy_config, socket_ops=None, proxy_uuid=None)
¶
Initialize the ZMQ Proxy. This is a base class for all ZMQ Proxies.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frontend_socket_class
|
type[BaseZMQClient]
|
The frontend socket class. |
required |
backend_socket_class
|
type[BaseZMQClient]
|
The backend socket class. |
required |
context
|
Context
|
The ZMQ context. |
required |
zmq_proxy_config
|
BaseZMQProxyConfig
|
The ZMQ proxy configuration. |
required |
socket_ops
|
dict
|
Additional socket options to set. |
None
|
proxy_uuid
|
str
|
An optional UUID for the proxy instance. If not provided, a new UUID will be generated. This is useful for tracing and debugging purposes. |
None
|
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | |
from_config(config, socket_ops=None)
abstractmethod
classmethod
¶
Create a BaseZMQProxy from a BaseZMQProxyConfig, or None if not provided.
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
149 150 151 152 153 154 155 156 157 | |
run()
async
¶
Start the Base ZMQ Proxy.
This method starts the proxy and waits for it to complete asynchronously. The proxy forwards messages between the frontend and backend sockets.
Raises:
| Type | Description |
|---|---|
ProxyError
|
If the proxy produces an error. |
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 | |
stop()
async
¶
Shutdown the BaseZMQProxy.
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 | |
ProxySocketClient
¶
Bases: BaseZMQClient
A ZMQ Proxy socket client class that extends BaseZMQClient.
This class is used to create proxy sockets for the frontend, backend, capture, and control endpoint types of a ZMQ Proxy.
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | |
ZMQProxyFactory
¶
Bases: FactoryMixin[ZMQProxyType, BaseZMQProxy]
A factory for creating ZMQ proxies. see :class:FactoryMixin for more details.
Source code in aiperf/common/comms/zmq/zmq_proxy_base.py
273 274 | |
aiperf.common.comms.zmq.zmq_proxy_sockets¶
ZMQDealerRouterProxy = define_proxy_class(ZMQProxyType.DEALER_ROUTER, create_proxy_socket_class(SocketType.ROUTER, ProxyEndType.Frontend), create_proxy_socket_class(SocketType.DEALER, ProxyEndType.Backend))
module-attribute
¶
A ROUTER socket for the proxy's frontend and a DEALER socket for the proxy's backend.
ASCII Diagram: ┌───────────┐ ┌──────────────────────────────────┐ ┌───────────┐ │ DEALER │<───>│ PROXY │<────>│ ROUTER │ │ Client 1 │ │ ┌──────────┐ ┌──────────┐ │ │ Service 1 │ └───────────┘ │ │ ROUTER │<─────> │ DEALER │ │ └───────────┘ ┌───────────┐ │ │ Frontend │ │ Backend │ │ ┌───────────┐ │ DEALER │<───>│ └──────────┘ └──────────┘ │<────>│ ROUTER │ │ Client N │ └──────────────────────────────────┘ │ Service N │ └───────────┘ └───────────┘
The ROUTER frontend socket receives messages from DEALER clients and forwards them through the proxy to ROUTER services. The ZMQ proxy handles the message routing automatically.
The DEALER backend socket receives messages from ROUTER services and forwards them through the proxy to DEALER clients. The ZMQ proxy handles the message routing automatically.
CRITICAL: This socket must NOT have an identity when used in a proxy configuration, as it needs to be transparent to preserve routing envelopes for proper response forwarding back to original DEALER clients.
ZMQPushPullProxy = define_proxy_class(ZMQProxyType.PUSH_PULL, create_proxy_socket_class(SocketType.PULL, ProxyEndType.Frontend), create_proxy_socket_class(SocketType.PUSH, ProxyEndType.Backend))
module-attribute
¶
A PULL socket for the proxy's frontend and a PUSH socket for the proxy's backend.
ASCII Diagram: ┌───────────┐ ┌─────────────────────────────────┐ ┌───────────┐ │ PUSH │─────>│ PROXY │─────>│ PULL │ │ Client 1 │ │ ┌──────────┐ ┌──────────┐ │ │ Service 1 │ └───────────┘ │ │ PULL │──────>│ PUSH │ │ └───────────┘ ┌───────────┐ │ │ Frontend │ │ Backend │ │ ┌───────────┐ │ PUSH │─────>│ └──────────┘ └──────────┘ │─────>│ PULL │ │ Client N │ └─────────────────────────────────┘ │ Service N │ └───────────┘ └───────────┘
The PULL frontend socket receives messages from PUSH clients and forwards them through the proxy to PUSH services. The ZMQ proxy handles the message routing automatically.
The PUSH backend socket forwards messages from the proxy to PULL services. The ZMQ proxy handles the message routing automatically.
ZMQXPubXSubProxy = define_proxy_class(ZMQProxyType.XPUB_XSUB, create_proxy_socket_class(SocketType.XSUB, ProxyEndType.Frontend), create_proxy_socket_class(SocketType.XPUB, ProxyEndType.Backend))
module-attribute
¶
An XSUB socket for the proxy's frontend and an XPUB socket for the proxy's backend.
ASCII Diagram: ┌───────────┐ ┌─────────────────────────────────┐ ┌───────────┐ │ PUB │───>│ PROXY │───>│ SUB │ │ Client 1 │ │ ┌──────────┐ ┌──────────┐ │ │ Service 1 │ └───────────┘ │ │ XSUB │──────>│ XPUB │ │ └───────────┘ ┌───────────┐ │ │ Frontend │ │ Backend │ │ ┌───────────┐ │ PUB │───>│ └──────────┘ └──────────┘ │───>│ SUB │ │ Client N │ └─────────────────────────────────┘ │ Service N │ └───────────┘ └───────────┘
The XSUB frontend socket receives messages from PUB clients and forwards them through the proxy to XPUB services. The ZMQ proxy handles the message routing automatically.
The XPUB backend socket forwards messages from the proxy to SUB services. The ZMQ proxy handles the message routing automatically.
create_proxy_socket_class(socket_type, end_type)
¶
Create a proxy socket class using the specified socket type. This is used to reduce the boilerplate code required to create a ZMQ Proxy class.
Source code in aiperf/common/comms/zmq/zmq_proxy_sockets.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | |
define_proxy_class(proxy_type, frontend_socket_class, backend_socket_class)
¶
This function reduces the boilerplate code required to create a ZMQ Proxy class. It will generate a ZMQ Proxy class and register it with the ZMQProxyFactory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
proxy_type
|
ZMQProxyType
|
The type of proxy to generate. |
required |
frontend_socket_class
|
type[BaseZMQClient]
|
The class of the frontend socket. |
required |
backend_socket_class
|
type[BaseZMQClient]
|
The class of the backend socket. |
required |
Source code in aiperf/common/comms/zmq/zmq_proxy_sockets.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | |
aiperf.common.config.audio_config¶
AudioConfig
¶
Bases: BaseConfig
A configuration class for defining audio related settings.
Source code in aiperf/common/config/audio_config.py
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | |
AudioLengthConfig
¶
Bases: BaseConfig
A configuration class for defining audio length related settings.
Source code in aiperf/common/config/audio_config.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | |
aiperf.common.config.base_config¶
BaseConfig
¶
Bases: AIPerfBaseModel
Base configuration class for all configurations.
Source code in aiperf/common/config/base_config.py
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | |
serialize_to_yaml(verbose=False, indent=4)
¶
Serialize a Pydantic model to a YAML string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
verbose
|
bool
|
Whether to include verbose comments in the YAML output. |
False
|
indent
|
int
|
The per-level indentation to use. |
4
|
Source code in aiperf/common/config/base_config.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | |
aiperf.common.config.config_defaults¶
aiperf.common.config.config_validators¶
parse_file(value)
¶
Parses the given string value and returns a Path object if the value represents a valid file or directory. Returns None if the input value is empty. Args: value (str): The string value to parse. Returns: Optional[Path]: A Path object if the value is valid, or None if the value is empty. Raises: ValueError: If the value is not a valid file or directory.
Source code in aiperf/common/config/config_validators.py
201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 | |
parse_goodput(goodputs)
¶
Parses and validates a dictionary of goodput values, ensuring that all values are non-negative integers or floats, and converts them to floats. Args: goodputs (Dict[str, Any]): A dictionary where keys are target metric names (strings) and values are the corresponding goodput values. Returns: Dict[str, float]: A dictionary with the same keys as the input, but with all values converted to floats. Raises: ValueError: If any value in the input dictionary is not an integer or float, or if any value is negative.
Source code in aiperf/common/config/config_validators.py
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 | |
parse_service_types(input)
¶
Parses the input to ensure it is a set of service types. Will replace hyphens with underscores for user convenience.
Source code in aiperf/common/config/config_validators.py
73 74 75 76 77 78 79 80 81 82 | |
parse_str_or_csv_list(input)
¶
Parses the input to ensure it is either a string or a list. If the input is a string, it splits the string by commas and trims any whitespace around each element, returning the result as a list. If the input is already a list, it will split each item by commas and trim any whitespace around each element, returning the combined result as a list. If the input is neither a string nor a list, a ValueError is raised.
[1, 2, 3] -> [1, 2, 3] "1,2,3" -> ["1", "2", "3"]["1,2,3", "4,5,6"] -> ["1", "2", "3", "4", "5", "6"]["1,2,3", 4, 5] -> ["1", "2", "3", 4, 5]
Source code in aiperf/common/config/config_validators.py
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
parse_str_or_dict(input)
¶
Parses the input to ensure it is a dictionary.
- If the input is a string:
- If the string starts with a '{', it is parsed as a JSON string.
- Otherwise, it splits the string by commas and then for each item, it splits the item by colons into key and value, trims any whitespace.
- If the input is already a dictionary, it is returned as-is.
- If the input is a list, it is converted to a dictionary by splitting each string by colons into key and value, trims any whitespace.
- Otherwise, a ValueError is raised.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input
|
Any
|
The input to be parsed. Expected to be a string, list, or dictionary. |
required |
Returns: dict[str, Any]: A dictionary derived from the input. Raises: ValueError: If the input is neither a string, list, nor dictionary, or if the parsing fails.
Source code in aiperf/common/config/config_validators.py
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | |
parse_str_or_list(input)
¶
Parses the input to ensure it is either a string or a list. If the input is a string, it splits the string by commas and trims any whitespace around each element, returning the result as a list. If the input is already a list, it is returned as-is. If the input is neither a string nor a list, a ValueError is raised. Args: input (Any): The input to be parsed. Expected to be a string or a list. Returns: list: A list of strings derived from the input. Raises: ValueError: If the input is neither a string nor a list.
Source code in aiperf/common/config/config_validators.py
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | |
parse_str_or_list_of_positive_values(input)
¶
Parses the input to ensure it is a list of positive integers or floats.
This function first converts the input into a list using parse_str_or_list.
It then validates that each value in the list is either an integer or a float
and that all values are strictly greater than zero. If any value fails this
validation, a ValueError is raised.
Args:
input (Any): The input to be parsed. It can be a string or a list.
Returns:
List[Any]: A list of positive integers or floats.
Raises:
ValueError: If any value in the parsed list is not a positive integer or float.
Source code in aiperf/common/config/config_validators.py
145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | |
aiperf.common.config.conversation_config¶
ConversationConfig
¶
Bases: BaseConfig
A configuration class for defining conversations related settings.
Source code in aiperf/common/config/conversation_config.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | |
TurnConfig
¶
Bases: BaseConfig
A configuration class for defining turn related settings in a conversation.
Source code in aiperf/common/config/conversation_config.py
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | |
TurnDelayConfig
¶
Bases: BaseConfig
A configuration class for defining turn delay related settings.
Source code in aiperf/common/config/conversation_config.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | |
aiperf.common.config.endpoint_config¶
EndPointConfig
¶
Bases: BaseConfig
A configuration class for defining endpoint related settings.
Source code in aiperf/common/config/endpoint_config.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | |
aiperf.common.config.image_config¶
ImageConfig
¶
Bases: BaseConfig
A configuration class for defining image related settings.
Source code in aiperf/common/config/image_config.py
86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | |
ImageHeightConfig
¶
Bases: BaseConfig
A configuration class for defining image height related settings.
Source code in aiperf/common/config/image_config.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
ImageWidthConfig
¶
Bases: BaseConfig
A configuration class for defining image width related settings.
Source code in aiperf/common/config/image_config.py
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | |
aiperf.common.config.input_config¶
InputConfig
¶
Bases: BaseConfig
A configuration class for defining input related settings.
Source code in aiperf/common/config/input_config.py
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | |
validate_fixed_schedule()
¶
Validate the fixed schedule configuration.
Source code in aiperf/common/config/input_config.py
34 35 36 37 38 39 40 41 42 | |
aiperf.common.config.loader¶
load_service_config()
¶
Load the service configuration.
Source code in aiperf/common/config/loader.py
7 8 9 10 | |
load_user_config()
¶
Load the user configuration.
Source code in aiperf/common/config/loader.py
13 14 15 16 | |
aiperf.common.config.loadgen_config¶
LoadGeneratorConfig
¶
Bases: BaseConfig
A configuration class for defining top-level load generator settings.
Source code in aiperf/common/config/loadgen_config.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | |
aiperf.common.config.measurement_config¶
MeasurementConfig
¶
Bases: BaseConfig
A configuration class for defining top-level measurement settings.
Source code in aiperf/common/config/measurement_config.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | |
aiperf.common.config.output_config¶
OutputConfig
¶
Bases: BaseConfig
A configuration class for defining output related settings.
Source code in aiperf/common/config/output_config.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | |
aiperf.common.config.prompt_config¶
InputTokensConfig
¶
Bases: BaseConfig
A configuration class for defining input token related settings.
Source code in aiperf/common/config/prompt_config.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
OutputTokensConfig
¶
Bases: BaseConfig
A configuration class for defining output token related settings.
Source code in aiperf/common/config/prompt_config.py
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | |
PrefixPromptConfig
¶
Bases: BaseConfig
A configuration class for defining prefix prompt related settings.
Source code in aiperf/common/config/prompt_config.py
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | |
PromptConfig
¶
Bases: BaseConfig
A configuration class for defining prompt related settings.
Source code in aiperf/common/config/prompt_config.py
178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 | |
aiperf.common.config.service_config¶
ServiceConfig
¶
Bases: BaseSettings
Base configuration for all services. It will be provided to all services during their init function.
Source code in aiperf/common/config/service_config.py
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 | |
validate_comm_config()
¶
Initialize the comm_config if it is not provided, based on the comm_backend.
Source code in aiperf/common/config/service_config.py
48 49 50 51 52 53 54 55 56 57 58 | |
validate_log_level_from_verbose_flags()
¶
Set log level based on verbose flags.
Source code in aiperf/common/config/service_config.py
39 40 41 42 43 44 45 46 | |
aiperf.common.config.sweep_config¶
SweepConfig
¶
Bases: BaseConfig
A sweep of parameters.
Source code in aiperf/common/config/sweep_config.py
99 100 | |
SweepParam
¶
Bases: BaseConfig
A parameter to be swept.
Source code in aiperf/common/config/sweep_config.py
8 9 | |
aiperf.common.config.tokenizer_config¶
TokenizerConfig
¶
Bases: BaseConfig
A configuration class for defining tokenizer related settings.
Source code in aiperf/common/config/tokenizer_config.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | |
aiperf.common.config.user_config¶
UserConfig
¶
Bases: BaseConfig
A configuration class for defining top-level user settings.
Source code in aiperf/common/config/user_config.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | |
aiperf.common.config.worker_config¶
WorkersConfig
¶
Bases: BaseConfig
Worker configuration.
Source code in aiperf/common/config/worker_config.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
aiperf.common.config.zmq_config¶
BaseZMQCommunicationConfig
¶
Bases: BaseModel, ABC
Configuration for ZMQ communication.
Source code in aiperf/common/config/zmq_config.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | |
credit_drop_address
abstractmethod
property
¶
Get the credit drop address based on protocol configuration.
credit_return_address
abstractmethod
property
¶
Get the credit return address based on protocol configuration.
records_push_pull_address
abstractmethod
property
¶
Get the inference push/pull address based on protocol configuration.
get_address(address_type)
¶
Get the actual address based on the address type.
Source code in aiperf/common/config/zmq_config.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | |
BaseZMQProxyConfig
¶
Bases: BaseModel, ABC
Configuration Protocol for ZMQ Proxy.
Source code in aiperf/common/config/zmq_config.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | |
backend_address
abstractmethod
property
¶
Get the backend address based on protocol configuration.
capture_address
abstractmethod
property
¶
Get the capture address based on protocol configuration.
control_address
abstractmethod
property
¶
Get the control address based on protocol configuration.
frontend_address
abstractmethod
property
¶
Get the frontend address based on protocol configuration.
ZMQIPCConfig
¶
Bases: BaseZMQCommunicationConfig
Configuration for IPC transport.
Source code in aiperf/common/config/zmq_config.py
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 | |
ZMQIPCProxyConfig
¶
Bases: BaseZMQProxyConfig
Configuration for IPC proxy.
Source code in aiperf/common/config/zmq_config.py
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | |
backend_address
property
¶
Get the backend address based on protocol configuration.
capture_address
property
¶
Get the capture address based on protocol configuration.
control_address
property
¶
Get the control address based on protocol configuration.
frontend_address
property
¶
Get the frontend address based on protocol configuration.
ZMQTCPConfig
¶
Bases: BaseZMQCommunicationConfig
Configuration for TCP transport.
Source code in aiperf/common/config/zmq_config.py
157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 | |
ZMQTCPProxyConfig
¶
Bases: BaseZMQProxyConfig
Configuration for TCP proxy.
Source code in aiperf/common/config/zmq_config.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | |
backend_address
property
¶
Get the backend address based on protocol configuration.
capture_address
property
¶
Get the capture address based on protocol configuration.
control_address
property
¶
Get the control address based on protocol configuration.
frontend_address
property
¶
Get the frontend address based on protocol configuration.
aiperf.common.constants¶
DEFAULT_COMMS_REQUEST_TIMEOUT = 10.0
module-attribute
¶
Default timeout for requests from req_clients to rep_clients in seconds.
TASK_CANCEL_TIMEOUT_LONG = 5.0
module-attribute
¶
Maximum time to wait for complex tasks to complete when cancelling them (like parent tasks).
TASK_CANCEL_TIMEOUT_SHORT = 2.0
module-attribute
¶
Maximum time to wait for simple tasks to complete when cancelling them.
aiperf.common.enums.base_enums¶
CaseInsensitiveStrEnum
¶
Bases: str, Enum
CaseInsensitiveStrEnum is a custom enumeration class that extends str and Enum to provide case-insensitive
lookup functionality for its members.
Source code in aiperf/common/enums/base_enums.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | |
aiperf.common.enums.benchmark_suite_enums¶
BenchmarkSuiteCompletionTrigger
¶
Bases: CaseInsensitiveStrEnum
Determines how the suite completion is determined in order to know how to track the progress.
Source code in aiperf/common/enums/benchmark_suite_enums.py
7 8 9 10 11 | |
COMPLETED_PROFILES = 'completed_profiles'
class-attribute
instance-attribute
¶
The suite will run until all profiles are completed.
BenchmarkSuiteType
¶
Bases: CaseInsensitiveStrEnum
Determines the type of suite to know how to track the progress.
Source code in aiperf/common/enums/benchmark_suite_enums.py
19 20 21 22 23 | |
SINGLE_PROFILE = 'single_profile'
class-attribute
instance-attribute
¶
A suite with a single profile run.
aiperf.common.enums.command_enums¶
CommandResponseStatus
¶
Bases: CaseInsensitiveStrEnum
Status of a command response.
Source code in aiperf/common/enums/command_enums.py
35 36 37 38 39 | |
CommandType
¶
Bases: CaseInsensitiveStrEnum
List of commands that the SystemController can send to component services.
Source code in aiperf/common/enums/command_enums.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | |
PROCESS_RECORDS = 'process_records'
class-attribute
instance-attribute
¶
A command sent to process records. This will process the records and return the services to their pre-record processing state.
PROFILE_CANCEL = 'profile_cancel'
class-attribute
instance-attribute
¶
A command sent to cancel a profile run. This will stop the current profile run and process the partial results.
PROFILE_CONFIGURE = 'profile_configure'
class-attribute
instance-attribute
¶
A command sent to configure a service in preparation for a profile run. This will override the current configuration.
PROFILE_START = 'profile_start'
class-attribute
instance-attribute
¶
A command sent to indicate that a service should begin profiling using the current configuration.
PROFILE_STOP = 'profile_stop'
class-attribute
instance-attribute
¶
A command sent to indicate that a service should stop doing profile related work, as the profile run is complete.
SHUTDOWN = 'shutdown'
class-attribute
instance-attribute
¶
A command sent to shutdown a service. This will stop the service gracefully no matter what state it is in.
aiperf.common.enums.communication_enums¶
CommunicationBackend
¶
Bases: CaseInsensitiveStrEnum
Supported communication backends.
Source code in aiperf/common/enums/communication_enums.py
7 8 9 10 11 12 13 14 | |
CommunicationClientAddressType
¶
Bases: CaseInsensitiveStrEnum
Enum for specifying the address type for communication clients. This is used to lookup the address in the communication config.
Source code in aiperf/common/enums/communication_enums.py
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | |
CREDIT_DROP = 'credit_drop'
class-attribute
instance-attribute
¶
Address to send CreditDrop messages from the TimingManager to the Worker.
CREDIT_RETURN = 'credit_return'
class-attribute
instance-attribute
¶
Address to send CreditReturn messages from the Worker to the TimingManager.
DATASET_MANAGER_PROXY_BACKEND = 'dataset_manager_proxy_backend'
class-attribute
instance-attribute
¶
Backend address for the DatasetManager to receive requests from clients.
DATASET_MANAGER_PROXY_FRONTEND = 'dataset_manager_proxy_frontend'
class-attribute
instance-attribute
¶
Frontend address for sending requests to the DatasetManager.
EVENT_BUS_PROXY_BACKEND = 'event_bus_proxy_backend'
class-attribute
instance-attribute
¶
Backend address for services to subscribe to messages.
EVENT_BUS_PROXY_FRONTEND = 'event_bus_proxy_frontend'
class-attribute
instance-attribute
¶
Frontend address for services to publish messages to.
RAW_INFERENCE_PROXY_BACKEND = 'raw_inference_proxy_backend'
class-attribute
instance-attribute
¶
Backend address for the InferenceParser to receive raw inference messages from Workers.
RAW_INFERENCE_PROXY_FRONTEND = 'raw_inference_proxy_frontend'
class-attribute
instance-attribute
¶
Frontend address for sending raw inference messages to the InferenceParser from Workers.
RECORDS = 'records'
class-attribute
instance-attribute
¶
Address to send parsed records from InferenceParser to RecordManager.
CommunicationClientType
¶
Bases: CaseInsensitiveStrEnum
Enum for specifying the communication client type for communication clients.
Source code in aiperf/common/enums/communication_enums.py
17 18 19 20 21 22 23 24 25 | |
ZMQProxyType
¶
Bases: CaseInsensitiveStrEnum
Types of ZMQ proxies.
Source code in aiperf/common/enums/communication_enums.py
60 61 62 63 64 65 | |
aiperf.common.enums.data_exporter_enums¶
aiperf.common.enums.dataset_enums¶
AudioFormat
¶
Bases: CaseInsensitiveStrEnum
Types of audio formats supported by AIPerf.
Source code in aiperf/common/enums/dataset_enums.py
34 35 36 37 38 | |
ComposerType
¶
Bases: CaseInsensitiveStrEnum
The type of composer to use for the dataset.
Source code in aiperf/common/enums/dataset_enums.py
7 8 9 10 11 12 13 14 | |
CustomDatasetType
¶
Bases: CaseInsensitiveStrEnum
Defines the type of JSONL custom dataset from the user.
Source code in aiperf/common/enums/dataset_enums.py
17 18 19 20 21 22 23 | |
ImageFormat
¶
Bases: CaseInsensitiveStrEnum
Types of image formats supported by AIPerf.
Source code in aiperf/common/enums/dataset_enums.py
26 27 28 29 30 31 | |
PromptSource
¶
Bases: CaseInsensitiveStrEnum
Source of prompts for the model.
Source code in aiperf/common/enums/dataset_enums.py
41 42 43 44 45 46 | |
aiperf.common.enums.endpoints_enums¶
EndpointType
¶
Bases: CaseInsensitiveStrEnum
Endpoint types.
These determine the format of request payload to send to the model.
Similar to endpoint_type_map and OutputFormat from genai-perf.
Source code in aiperf/common/enums/endpoints_enums.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | |
endpoint_path()
¶
Get the endpoint path for the endpoint type.
Source code in aiperf/common/enums/endpoints_enums.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | |
response_payload_type()
¶
Get the response payload type for the request payload type.
Source code in aiperf/common/enums/endpoints_enums.py
67 68 69 | |
ResponsePayloadType
¶
Bases: CaseInsensitiveStrEnum
Response payload types.
These determine the format of the response payload that the model will return.
Equivalent to output_format from genai-perf.
Source code in aiperf/common/enums/endpoints_enums.py
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | |
from_endpoint_type(endpoint_type)
classmethod
¶
Get the response payload type for the endpoint type.
Source code in aiperf/common/enums/endpoints_enums.py
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | |
aiperf.common.enums.logging_enums¶
AIPerfLogLevel
¶
Bases: CaseInsensitiveStrEnum
Log levels for AIPerfLogger.
Source code in aiperf/common/enums/logging_enums.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | |
level
property
¶
Get the integer level equivalent.
aiperf.common.enums.measurement_enums¶
aiperf.common.enums.message_enums¶
MessageType
¶
Bases: CaseInsensitiveStrEnum
The various types of messages that can be sent between services.
The message type is used to determine what Pydantic model the message maps to,
based on the message_type field in the message model. For detailed explanations
of each message type, go to its definition in :mod:aiperf.common.messages.
Source code in aiperf/common/enums/message_enums.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | |
NotificationType
¶
Bases: CaseInsensitiveStrEnum
Types of notifications that can be sent to other services.
Source code in aiperf/common/enums/message_enums.py
55 56 57 58 59 | |
DATASET_CONFIGURED = 'dataset_configured'
class-attribute
instance-attribute
¶
A notification sent to notify other services that the dataset has been configured.
aiperf.common.enums.metric_enums¶
MetricTimeType
¶
Bases: CaseInsensitiveStrEnum
Defines the time types for metrics.
Source code in aiperf/common/enums/metric_enums.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | |
short_name()
¶
Get the short name for the time type.
Source code in aiperf/common/enums/metric_enums.py
16 17 18 19 20 21 22 23 | |
aiperf.common.enums.model_enums¶
Modality
¶
Bases: CaseInsensitiveStrEnum
Modality of the model. Can be used to determine the type of data to send to the model in conjunction with the ModelSelectionStrategy.MODALITY_AWARE.
Source code in aiperf/common/enums/model_enums.py
7 8 9 10 11 12 13 14 15 16 | |
ModelSelectionStrategy
¶
Bases: CaseInsensitiveStrEnum
Strategy for selecting the model to use for the request.
Source code in aiperf/common/enums/model_enums.py
19 20 21 22 23 24 | |
aiperf.common.enums.post_processor_enums¶
StreamingPostProcessorType
¶
Bases: CaseInsensitiveStrEnum
Type of response streamer.
Source code in aiperf/common/enums/post_processor_enums.py
11 12 13 14 15 16 17 18 19 20 21 | |
BASIC_METRICS = 'basic_metrics'
class-attribute
instance-attribute
¶
Streamer that handles the basic metrics of the records.
JSONL = 'jsonl'
class-attribute
instance-attribute
¶
Streams all parsed records to a JSONL file.
PROCESSING_STATS = 'processing_stats'
class-attribute
instance-attribute
¶
Streamer that provides the processing stats of the records.
aiperf.common.enums.service_enums¶
ServiceRegistrationStatus
¶
Bases: CaseInsensitiveStrEnum
Defines the various states a service can be in during registration with the SystemController.
Source code in aiperf/common/enums/service_enums.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | |
ERROR = 'error'
class-attribute
instance-attribute
¶
The service registration failed.
REGISTERED = 'registered'
class-attribute
instance-attribute
¶
The service is registered with the SystemController.
TIMEOUT = 'timeout'
class-attribute
instance-attribute
¶
The service registration timed out.
UNREGISTERED = 'unregistered'
class-attribute
instance-attribute
¶
The service is not registered with the SystemController. This is the initial state.
WAITING = 'waiting'
class-attribute
instance-attribute
¶
The service is waiting for the SystemController to register it. This is a temporary state that should be followed by REGISTERED, TIMEOUT, or ERROR.
ServiceRunType
¶
Bases: CaseInsensitiveStrEnum
The different ways the SystemController should run the component services.
Source code in aiperf/common/enums/service_enums.py
7 8 9 10 11 12 13 14 15 16 | |
KUBERNETES = 'k8s'
class-attribute
instance-attribute
¶
Run each service as a separate Kubernetes pod. This is the default way for multi-node deployments.
MULTIPROCESSING = 'process'
class-attribute
instance-attribute
¶
Run each service as a separate process. This is the default way for single-node deployments.
ServiceState
¶
Bases: CaseInsensitiveStrEnum
States a service can be in throughout its lifecycle.
Source code in aiperf/common/enums/service_enums.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | |
CONFIGURING = 'configuring'
class-attribute
instance-attribute
¶
The service is currently configuring. This is a temporary state that should be followed by READY.
ERROR = 'error'
class-attribute
instance-attribute
¶
The service is currently in an error state.
INITIALIZING = 'initializing'
class-attribute
instance-attribute
¶
The service is currently initializing. This is a temporary state that should be followed by PENDING.
PENDING = 'pending'
class-attribute
instance-attribute
¶
The service is pending configuration.
READY = 'ready'
class-attribute
instance-attribute
¶
The service has been configured and is ready to be started.
RUNNING = 'running'
class-attribute
instance-attribute
¶
The service is running.
SHUTDOWN = 'shutdown'
class-attribute
instance-attribute
¶
The service has been shutdown.
STARTING = 'starting'
class-attribute
instance-attribute
¶
The service is starting. This is a temporary state that should be followed by RUNNING.
STOPPED = 'stopped'
class-attribute
instance-attribute
¶
The service has been stopped.
STOPPING = 'stopping'
class-attribute
instance-attribute
¶
The service is stopping. This is a temporary state that should be followed by STOPPED.
UNKNOWN = 'unknown'
class-attribute
instance-attribute
¶
The service is in an unknown state.
ServiceType
¶
Bases: CaseInsensitiveStrEnum
Types of services in the AIPerf system.
This is used to identify the service type when registering with the SystemController. It can also be used for tracking purposes if multiple instances of the same service type are running.
Source code in aiperf/common/enums/service_enums.py
60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | |
aiperf.common.enums.sse_enums¶
SSEEventType
¶
Bases: CaseInsensitiveStrEnum
Event types in an SSE message. Many of these are custom and not defined by the SSE spec.
Source code in aiperf/common/enums/sse_enums.py
17 18 19 20 21 | |
SSEFieldType
¶
Bases: CaseInsensitiveStrEnum
Field types in an SSE message.
Source code in aiperf/common/enums/sse_enums.py
7 8 9 10 11 12 13 14 | |
aiperf.common.enums.system_enums¶
SystemState
¶
Bases: CaseInsensitiveStrEnum
State of the system as a whole.
This is used to track the state of the system as a whole, and is used to determine what actions to take when a signal is received.
Source code in aiperf/common/enums/system_enums.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | |
CONFIGURING = 'configuring'
class-attribute
instance-attribute
¶
The system is configuring services.
INITIALIZING = 'initializing'
class-attribute
instance-attribute
¶
The system is initializing. This is the initial state.
PROCESSING = 'processing'
class-attribute
instance-attribute
¶
The system is processing results.
PROFILING = 'profiling'
class-attribute
instance-attribute
¶
The system is running a profiling run.
READY = 'ready'
class-attribute
instance-attribute
¶
The system is ready to start profiling. This is a temporary state that should be followed by PROFILING.
SHUTDOWN = 'shutdown'
class-attribute
instance-attribute
¶
The system is shutting down. This is the final state.
STOPPING = 'stopping'
class-attribute
instance-attribute
¶
The system is stopping.
aiperf.common.enums.timing_enums¶
CreditPhase
¶
Bases: CaseInsensitiveStrEnum
The type of credit phase. This is used to identify which phase of the benchmark the credit is being used in, for tracking and reporting purposes.
Source code in aiperf/common/enums/timing_enums.py
30 31 32 33 34 35 36 37 38 39 40 | |
PROFILING = 'profiling'
class-attribute
instance-attribute
¶
The credit phase is the steady state phase. This is the primary phase of the benchmark, and what is used to calculate the final results.
WARMUP = 'warmup'
class-attribute
instance-attribute
¶
The credit phase is the warmup phase. This is used to warm up the model before the benchmark starts.
RequestRateMode
¶
Bases: CaseInsensitiveStrEnum
The different ways the RequestRateStrategy should generate requests.
Source code in aiperf/common/enums/timing_enums.py
20 21 22 23 24 25 26 27 | |
TimingMode
¶
Bases: CaseInsensitiveStrEnum
The different ways the TimingManager should generate requests.
Source code in aiperf/common/enums/timing_enums.py
7 8 9 10 11 12 13 14 15 16 17 | |
CONCURRENCY = 'concurrency'
class-attribute
instance-attribute
¶
A mode where the TimingManager will maintain a continuous stream of concurrent requests.
FIXED_SCHEDULE = 'fixed_schedule'
class-attribute
instance-attribute
¶
A mode where the TimingManager will send requests according to a fixed schedule.
REQUEST_RATE = 'request_rate'
class-attribute
instance-attribute
¶
A mode where the TimingManager will send requests at either a constant request rate or based on a poisson distribution.
aiperf.common.exceptions¶
AIPerfError
¶
Bases: Exception
Base class for all exceptions raised by AIPerf.
Source code in aiperf/common/exceptions.py
7 8 9 10 11 12 13 14 15 16 | |
__str__()
¶
Return the string representation of the exception with the class name.
Source code in aiperf/common/exceptions.py
14 15 16 | |
raw_str()
¶
Return the raw string representation of the exception.
Source code in aiperf/common/exceptions.py
10 11 12 | |
AIPerfMultiError
¶
Bases: AIPerfError
Exception raised when running multiple tasks and one or more fail.
Source code in aiperf/common/exceptions.py
19 20 21 22 23 24 25 26 27 | |
CommunicationError
¶
Bases: AIPerfError
Generic communication error.
Source code in aiperf/common/exceptions.py
70 71 | |
ConfigurationError
¶
Bases: AIPerfError
Exception raised when something fails to configure, or there is a configuration error.
Source code in aiperf/common/exceptions.py
50 51 | |
DatasetError
¶
Bases: AIPerfError
Generic dataset error.
Source code in aiperf/common/exceptions.py
74 75 | |
DatasetGeneratorError
¶
Bases: AIPerfError
Generic dataset generator error.
Source code in aiperf/common/exceptions.py
78 79 | |
FactoryCreationError
¶
Bases: AIPerfError
Exception raised when a factory encounters an error while creating a class.
Source code in aiperf/common/exceptions.py
94 95 | |
InferenceClientError
¶
Bases: AIPerfError
Exception raised when a inference client encounters an error.
Source code in aiperf/common/exceptions.py
82 83 | |
InitializationError
¶
Bases: AIPerfError
Exception raised when something fails to initialize.
Source code in aiperf/common/exceptions.py
46 47 | |
InvalidPayloadError
¶
Bases: InferenceClientError
Exception raised when a inference client receives an invalid payload.
Source code in aiperf/common/exceptions.py
86 87 | |
InvalidStateError
¶
Bases: AIPerfError
Exception raised when something is in an invalid state.
Source code in aiperf/common/exceptions.py
58 59 | |
MetricTypeError
¶
Bases: AIPerfError
Exception raised when a metric type encounters an error while creating a class.
Source code in aiperf/common/exceptions.py
98 99 | |
NotFoundError
¶
Bases: AIPerfError
Exception raised when something is not found or not available.
Source code in aiperf/common/exceptions.py
66 67 | |
NotInitializedError
¶
Bases: AIPerfError
Exception raised when something that should be initialized is not.
Source code in aiperf/common/exceptions.py
54 55 | |
ProxyError
¶
Bases: AIPerfError
Exception raised when a proxy encounters an error.
Source code in aiperf/common/exceptions.py
106 107 | |
ServiceError
¶
Bases: AIPerfError
Generic service error.
Source code in aiperf/common/exceptions.py
30 31 32 33 34 35 36 37 38 39 40 41 42 43 | |
ShutdownError
¶
Bases: AIPerfError
Exception raised when a service encounters an error while shutting down.
Source code in aiperf/common/exceptions.py
102 103 | |
UnsupportedHookError
¶
Bases: AIPerfError
Exception raised when a hook is defined on a class that does not support it.
Source code in aiperf/common/exceptions.py
90 91 | |
ValidationError
¶
Bases: AIPerfError
Exception raised when something fails validation.
Source code in aiperf/common/exceptions.py
62 63 | |
aiperf.common.factories¶
ComposerFactory
¶
Bases: FactoryMixin['ComposerType', 'BaseDatasetComposer']
Factory for registering and creating BaseDatasetComposer instances based on the specified composer type.
Example:
# Register a new composer type
@ComposerFactory.register(ComposerType.SYNTHETIC)
class SyntheticDatasetComposer(BaseDatasetComposer):
pass
# Create a new composer instance
composer = ComposerFactory.create_instance(
ComposerType.SYNTHETIC,
config=InputConfig(
conversation=ConversationConfig(num=10),
prompt=PromptConfig(batch_size=10),
)
)
Source code in aiperf/common/factories.py
275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 | |
CustomDatasetFactory
¶
Bases: FactoryMixin['CustomDatasetType', 'CustomDatasetLoaderProtocol']
Factory for registering and creating CustomDatasetLoader instances based on the specified custom dataset type.
Example:
# Register a new custom dataset type
@CustomDatasetFactory.register(CustomDatasetType.MOONCAKE_TRACE)
class MooncakeTraceDatasetLoader(CustomDatasetLoader):
pass
# Create a new custom dataset loader instance
custom_dataset_loader = CustomDatasetFactory.create_instance(
CustomDatasetType.MOONCAKE_TRACE, **kwargs
)
Source code in aiperf/common/factories.py
298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 | |
DataExporterFactory
¶
Bases: FactoryMixin['DataExporterType', 'DataExporterProtocol']
Factory for registering and creating DataExporterInterface instances.
Example:
# Iterate over all registered data exporter types
for exporter_class in DataExporterFactory.get_all_classes():
exporter = exporter_class(endpoint_config)
exporter.export()
Source code in aiperf/common/factories.py
244 245 246 247 248 249 250 251 252 253 254 255 | |
FactoryMixin
¶
Bases: Generic[ClassEnumT, ClassProtocolT]
Defines a mixin for all factories, which supports registering and creating instances of classes.
This mixin is used to create a factory for a given class type and protocol.
Example:
# Define a new enum for the expected implementation types
# This is optional, but recommended for type safety.
class DatasetLoaderType(CaseInsensitiveStrEnum):
FILE = "file"
S3 = "s3"
# Define a new class protocol.
class DatasetLoaderProtocol(Protocol):
def load(self) -> Dataset:
pass
# Create a new factory for a given class type and protocol.
class DatasetFactory(FactoryMixin[DatasetLoaderType, DatasetLoaderProtocol]):
pass
# Register a new class type mapping to its corresponding class. It should implement the class protocol.
@DatasetFactory.register(DatasetLoaderType.FILE)
class FileDatasetLoader:
def __init__(self, filename: str):
self.filename = filename
def load(self) -> Dataset:
return Dataset.from_file(self.filename)
DatasetConfig = {
"type": DatasetLoaderType.FILE,
"filename": "data.csv"
}
# Create a new instance of the class.
if DatasetConfig["type"] == DatasetLoaderType.FILE:
dataset_instance = DatasetFactory.create_instance(DatasetLoaderType.FILE, filename=DatasetConfig["filename"])
else:
raise ValueError(f"Unsupported dataset loader type: {DatasetConfig['type']}")
dataset_instance.load()
Source code in aiperf/common/factories.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 | |
create_instance(class_type, **kwargs)
classmethod
¶
Create a new class instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
class_type
|
ClassEnumT | str
|
The type of class to create |
required |
**kwargs
|
Any
|
Additional arguments for the class |
{}
|
Returns:
| Type | Description |
|---|---|
ClassProtocolT
|
The created class instance |
Raises:
| Type | Description |
|---|---|
FactoryCreationError
|
If the class type is not registered or there is an error creating the instance |
Source code in aiperf/common/factories.py
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | |
get_all_class_types()
classmethod
¶
Get all registered class types.
Source code in aiperf/common/factories.py
203 204 205 206 | |
get_all_classes()
classmethod
¶
Get all registered classes.
Returns:
| Type | Description |
|---|---|
list[type[ClassProtocolT]]
|
A list of all registered class types implementing the expected protocol |
Source code in aiperf/common/factories.py
194 195 196 197 198 199 200 201 | |
get_all_classes_and_types()
classmethod
¶
Get all registered classes and their corresponding class types.
Source code in aiperf/common/factories.py
208 209 210 211 212 213 | |
get_class_from_type(class_type)
classmethod
¶
Get the class from a class type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
class_type
|
ClassEnumT | str
|
The class type to get the class from |
required |
Returns:
| Type | Description |
|---|---|
type[ClassProtocolT]
|
The class for the given class type |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the class type is not registered |
Source code in aiperf/common/factories.py
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | |
register(class_type, override_priority=0)
classmethod
¶
Register a new class type mapping to its corresponding class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
class_type
|
ClassEnumT | str
|
The type of class to register |
required |
override_priority
|
int
|
The priority of the override. The higher the priority, the more precedence the override has when multiple classes are registered for the same class type. Built-in classes have a priority of 0. |
0
|
Returns:
| Type | Description |
|---|---|
Callable
|
Decorator for the class that implements the class protocol |
Source code in aiperf/common/factories.py
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | |
register_all(*class_types, override_priority=0)
classmethod
¶
Register multiple class types mapping to a single corresponding class. This is useful if a single class implements multiple types. Currently only supports registering as a single override priority for all types.
Source code in aiperf/common/factories.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 | |
PostProcessorFactory
¶
Bases: FactoryMixin['PostProcessorType', 'PostProcessorProtocol']
Factory for registering and creating PostProcessor instances based on the specified post-processor type.
Example: ```python # Register a new post-processor type @PostProcessorFactory.register(PostProcessorType.METRIC_SUMMARY) class MetricSummary: pass
# Create a new post-processor instance
post_processor = PostProcessorFactory.create_instance(
PostProcessorType.METRIC_SUMMARY,
)
Source code in aiperf/common/factories.py
258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 | |
ServiceFactory
¶
Bases: FactoryMixin[ServiceType, 'BaseService']
Factory for registering and creating BaseService instances based on the specified service type.
Example:
# Register a new service type
@ServiceFactory.register(ServiceType.DATASET_MANAGER)
class DatasetManager(BaseService):
pass
# Create a new service instance in a separate process
service_class = ServiceFactory.get_class_from_type(service_type)
process = Process(
target=bootstrap_and_run_service,
name=f"{service_type}_process",
args=(service_class, self.config),
daemon=False,
)
Source code in aiperf/common/factories.py
221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 | |
StreamingPostProcessorFactory
¶
Bases: FactoryMixin[StreamingPostProcessorType, 'StreamingPostProcessor']
Factory for creating StreamingPostProcessor instances.
see: :class:FactoryMixin for more details.
Source code in aiperf/common/factories.py
320 321 322 323 324 325 | |
aiperf.common.hooks¶
This module provides an extensive hook system for AIPerf. It is designed to be used as a mixin for classes that support hooks. It provides a simple interface for registering and running hooks.
Classes should inherit from the :class:HooksMixin, and specify the supported
hook types by decorating the class with the :func:supports_hooks decorator.
The hook functions are registered by decorating functions with the various hook
decorators such as :func:on_init, :func:on_start, :func:on_stop, etc.
The hooks are run by calling the :meth:HooksMixin.run_hooks or
:meth:HooksMixin.run_hooks_async methods on the class.
More than one hook can be registered for a given hook type, and classes that inherit from classes with existing hooks will inherit the hooks from the base classes as well.
AIPERF_HOOK_TYPE = '__aiperf_hook_type__'
module-attribute
¶
Constant attribute name that marks a function's hook type.
HookType = AIPerfHook | AIPerfTaskHook | str
module-attribute
¶
Type alias for valid hook types. This is a union of the AIPerfHook enum, the AIPerfTaskHook enum, and any user-defined custom strings.
AIPerfHook
¶
Bases: CaseInsensitiveStrEnum
Enum for the various AIPerf hooks.
Note: If you add a new hook, you must also add it to the @supports_hooks decorator of the class you wish to use the hook in.
Source code in aiperf/common/hooks.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | |
AIPerfTaskHook
¶
Bases: CaseInsensitiveStrEnum
Enum for the various AIPerf task hooks.
Source code in aiperf/common/hooks.py
58 59 60 61 62 63 | |
HookSystem
¶
System for managing hooks.
This class is responsible for managing the hooks for a class. It will store the hooks in a dictionary, and provide methods to register and run the hooks.
Source code in aiperf/common/hooks.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 | |
__init__(supported_hooks)
¶
Initialize the hook system.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
supported_hooks
|
set[HookType]
|
The hook types that the class supports. |
required |
Source code in aiperf/common/hooks.py
88 89 90 91 92 93 94 95 96 97 | |
get_hooks(hook_type)
¶
Get all the registered hooks for the given hook type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hook_type
|
HookType
|
The hook type to get the hooks for. |
required |
Returns:
| Type | Description |
|---|---|
list[Callable]
|
A list of the hooks for the given hook type. |
Source code in aiperf/common/hooks.py
111 112 113 114 115 116 117 118 119 120 | |
register_hook(hook_type, func)
¶
Register a hook function for a given hook type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hook_type
|
HookType
|
The hook type to register the function for. |
required |
func
|
Callable
|
The function to register. |
required |
Source code in aiperf/common/hooks.py
99 100 101 102 103 104 105 106 107 108 109 | |
run_hooks(hook_type, *args, **kwargs)
async
¶
Run all the hooks for a given hook type serially. This will wait for each hook to complete before running the next one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hook_type
|
HookType
|
The hook type to run. |
required |
*args
|
The arguments to pass to the hooks. |
()
|
|
**kwargs
|
The keyword arguments to pass to the hooks. |
{}
|
Source code in aiperf/common/hooks.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 | |
run_hooks_async(hook_type, *args, **kwargs)
async
¶
Run all the hooks for a given hook type concurrently. This will run all the hooks at the same time and return when all the hooks have completed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hook_type
|
HookType
|
The hook type to run. |
required |
*args
|
The arguments to pass to the hooks. |
()
|
|
**kwargs
|
The keyword arguments to pass to the hooks. |
{}
|
Source code in aiperf/common/hooks.py
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 | |
aiperf_auto_task(interval)
¶
Decorator to indicate that the function is a task function. It will be started
and stopped automatically by the base class lifecycle.
See :func:aiperf.common.hooks.hook_decorator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interval
|
float
|
The interval in seconds to sleep between runs. |
required |
Source code in aiperf/common/hooks.py
317 318 319 320 321 322 323 324 325 326 327 328 329 330 | |
aiperf_task(func)
¶
Decorator to indicate that the function is a task function. It will be started
and stopped automatically by the base class lifecycle.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
307 308 309 310 311 312 313 314 | |
hook_decorator(hook_type, func)
¶
Generic decorator to specify that the function should be called during a specific hook.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hook_type
|
HookType
|
The hook type to decorate the function with. |
required |
func
|
Callable
|
The function to decorate. |
required |
Returns: The decorated function.
Source code in aiperf/common/hooks.py
226 227 228 229 230 231 232 233 234 235 236 237 | |
on_cleanup(func)
¶
Decorator to specify that the function should be called during cleanup.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
269 270 271 272 | |
on_configure(func)
¶
Decorator to specify that the function should be called during the service configuration.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
263 264 265 266 | |
on_init(func)
¶
Decorator to specify that the function should be called during initialization.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
245 246 247 248 | |
on_profile_configure(func)
¶
Decorator to specify that the function should be called during the service profile configuration.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
289 290 291 292 | |
on_profile_start(func)
¶
Decorator to specify that the function should be called during the service profile start.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
295 296 297 298 | |
on_profile_stop(func)
¶
Decorator to specify that the function should be called during the service profile stop.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
301 302 303 304 | |
on_run(func)
¶
Decorator to specify that the function should be called during run.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
275 276 277 278 | |
on_set_state(func)
¶
Decorator to specify that the function should be called when the service state is set.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
281 282 283 284 285 286 | |
on_start(func)
¶
Decorator to specify that the function should be called during start.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
251 252 253 254 | |
on_stop(func)
¶
Decorator to specify that the function should be called during stop.
See :func:aiperf.common.hooks.hook_decorator.
Source code in aiperf/common/hooks.py
257 258 259 260 | |
supports_hooks(*supported_hook_types)
¶
Decorator to indicate that a class supports hooks and sets the supported hook types.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
supported_hook_types
|
HookType
|
The hook types that the class supports. |
()
|
Returns:
| Type | Description |
|---|---|
Callable[[type], type]
|
The decorated class |
Source code in aiperf/common/hooks.py
186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 | |
aiperf.common.interfaces¶
DataExporterProtocol
¶
Bases: Protocol
Protocol for data exporters.
Any class implementing this protocol must provide an export method
that takes a list of Record objects and handles exporting them appropriately.
Source code in aiperf/common/interfaces.py
14 15 16 17 18 19 20 21 22 23 24 | |
export()
async
¶
Export the data.
Source code in aiperf/common/interfaces.py
22 23 24 | |
PostProcessorProtocol
¶
Bases: Protocol
PostProcessorProtocol is a protocol that defines the API for post-processors.
It requires an process method that takes a list of records and returns a result.
Source code in aiperf/common/interfaces.py
30 31 32 33 34 35 36 37 38 39 40 41 42 43 | |
process(records)
¶
Execute the post-processing logic on the given payload.
:param payload: The input data to be processed. :return: The processed data as a dictionary.
Source code in aiperf/common/interfaces.py
36 37 38 39 40 41 42 43 | |
ResponseExtractor
¶
Bases: Protocol
Base class for all response extractors.
Source code in aiperf/common/interfaces.py
51 52 53 54 55 56 57 58 | |
extract_response_data(record)
async
¶
Extract the text from a server response message.
Source code in aiperf/common/interfaces.py
54 55 56 57 58 | |
aiperf.common.logging¶
MultiProcessLogHandler
¶
Bases: RichHandler
Custom logging handler that forwards log records to a multiprocessing queue.
Source code in aiperf/common/logging.py
169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | |
emit(record)
¶
Emit a log record to the queue.
Source code in aiperf/common/logging.py
179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | |
create_file_handler(log_folder, level)
¶
Configure a file handler for logging.
Source code in aiperf/common/logging.py
149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | |
get_global_log_queue()
cached
¶
Get the global log queue. Will create a new queue if it doesn't exist.
Source code in aiperf/common/logging.py
23 24 25 26 | |
setup_child_process_logging(log_queue=None, service_id=None, service_config=None, user_config=None)
¶
Set up logging for a child process to send logs to the main process.
This should be called early in child process initialization.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
log_queue
|
Queue | None
|
The multiprocessing queue to send logs to. If None, tries to get the global queue. |
None
|
service_id
|
str | None
|
The ID of the service to log under. If None, logs will be under the process name. |
None
|
service_config
|
ServiceConfig | None
|
The service configuration used to determine the log level. |
None
|
user_config
|
UserConfig | None
|
The user configuration used to determine the log folder. |
None
|
Source code in aiperf/common/logging.py
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | |
setup_rich_logging(user_config, service_config)
¶
Set up rich logging with appropriate configuration.
Source code in aiperf/common/logging.py
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | |
aiperf.common.messages.base_messages¶
ErrorMessage
¶
Bases: Message
Message containing error data.
Source code in aiperf/common/messages/base_messages.py
118 119 120 121 122 123 | |
Message
¶
Bases: ExcludeIfNoneMixin
Base message class for optimized message handling.
This class provides a base for all messages, including common fields like message_type, request_ns, and request_id. It also supports optional field exclusion based on the @exclude_if_none decorator.
Each message model should inherit from this class, set the message_type field, and define its own additional fields. Optionally, the @exclude_if_none decorator can be used to specify which fields should be excluded from the serialized message if they are None.
Example:
@exclude_if_none(["some_field"])
class ExampleMessage(Message):
some_field: int | None = Field(default=None)
other_field: int = Field(default=1)
Source code in aiperf/common/messages/base_messages.py
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | |
from_json(json_str)
classmethod
¶
Fast deserialization without full validation
Source code in aiperf/common/messages/base_messages.py
89 90 91 92 93 94 95 96 97 98 99 100 101 102 | |
to_json()
¶
Fast serialization without full validation
Source code in aiperf/common/messages/base_messages.py
104 105 106 | |
RequiresRequestNSMixin
¶
Bases: Message
Mixin for messages that require a request_ns field.
Source code in aiperf/common/messages/base_messages.py
109 110 111 112 113 114 115 | |
aiperf.common.messages.command_messages¶
CommandMessage
¶
Bases: BaseServiceMessage
Message containing command data. This message is sent by the system controller to a service to command it to do something.
Source code in aiperf/common/messages/command_messages.py
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | |
CommandResponseMessage
¶
Bases: BaseServiceMessage
Message containing a command response. This message is sent by a component service to the system controller to respond to a command.
Source code in aiperf/common/messages/command_messages.py
70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | |
ProcessRecordsCommandData
¶
Bases: BaseModel
Data to send with the process records command.
Source code in aiperf/common/messages/command_messages.py
24 25 26 27 28 29 30 | |
aiperf.common.messages.credit_messages¶
CreditDropMessage
¶
Bases: BaseServiceMessage
Message indicating that a credit has been dropped. This message is sent by the timing manager to workers to indicate that credit(s) have been dropped.
Source code in aiperf/common/messages/credit_messages.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | |
CreditPhaseCompleteMessage
¶
Bases: BaseServiceMessage
Message for credit phase complete. Sent by the TimingManager to report that a credit phase has completed.
Source code in aiperf/common/messages/credit_messages.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | |
CreditPhaseProgressMessage
¶
Bases: BaseServiceMessage
Sent by the TimingManager to report the progress of a credit phase.
Source code in aiperf/common/messages/credit_messages.py
84 85 86 87 88 89 90 91 92 93 94 95 | |
CreditPhaseSendingCompleteMessage
¶
Bases: BaseServiceMessage
Message for credit phase sending complete. Sent by the TimingManager to report that a credit phase has completed sending.
Source code in aiperf/common/messages/credit_messages.py
98 99 100 101 102 103 104 105 106 107 108 | |
CreditPhaseStartMessage
¶
Bases: BaseServiceMessage
Message for credit phase start. Sent by the TimingManager to report that a credit phase has started.
Source code in aiperf/common/messages/credit_messages.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | |
CreditReturnMessage
¶
Bases: BaseServiceMessage
Message indicating that a credit has been returned. This message is sent by a worker to the timing manager to indicate that work has been completed.
Source code in aiperf/common/messages/credit_messages.py
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | |
CreditsCompleteMessage
¶
Bases: BaseServiceMessage
Credits complete message sent by the TimingManager to the System controller to signify all Credit Phases have been completed.
Source code in aiperf/common/messages/credit_messages.py
129 130 131 132 133 | |
aiperf.common.messages.dataset_messages¶
ConversationRequestMessage
¶
Bases: BaseServiceMessage
Message to request a full conversation by ID.
Source code in aiperf/common/messages/dataset_messages.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 | |
ConversationResponseMessage
¶
Bases: BaseServiceMessage
Message containing a full conversation.
Source code in aiperf/common/messages/dataset_messages.py
29 30 31 32 33 34 35 | |
ConversationTurnRequestMessage
¶
Bases: BaseServiceMessage
Message to request a single turn from a conversation.
Source code in aiperf/common/messages/dataset_messages.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | |
ConversationTurnResponseMessage
¶
Bases: BaseServiceMessage
Message containing a single turn from a conversation.
Source code in aiperf/common/messages/dataset_messages.py
56 57 58 59 60 61 62 63 | |
DatasetConfiguredNotification
¶
Bases: BaseServiceMessage
Notification sent to notify other services that the dataset has been configured.
Source code in aiperf/common/messages/dataset_messages.py
87 88 89 90 91 92 | |
DatasetTimingRequest
¶
Bases: BaseServiceMessage
Message for a dataset timing request.
Source code in aiperf/common/messages/dataset_messages.py
66 67 68 69 70 71 | |
DatasetTimingResponse
¶
Bases: BaseServiceMessage
Message for a dataset timing response.
Source code in aiperf/common/messages/dataset_messages.py
74 75 76 77 78 79 80 81 82 83 84 | |
aiperf.common.messages.health_messages¶
WorkerHealthMessage
¶
Bases: BaseServiceMessage
Message for a worker health check.
Source code in aiperf/common/messages/health_messages.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | |
completed_tasks
property
¶
The number of tasks that have been completed by the worker.
error_rate
property
¶
The error rate of the worker.
failed_tasks
property
¶
The number of tasks that have failed by the worker.
in_progress_tasks
property
¶
The number of tasks that are currently in progress by the worker.
total_tasks
property
¶
The total number of tasks that have been sent to the worker.
aiperf.common.messages.inference_messages¶
InferenceResultsMessage
¶
Bases: BaseServiceMessage
Message for a inference results.
Source code in aiperf/common/messages/inference_messages.py
20 21 22 23 24 25 26 27 | |
ParsedInferenceResultsMessage
¶
Bases: BaseServiceMessage
Message for a parsed inference results.
Source code in aiperf/common/messages/inference_messages.py
30 31 32 33 34 35 36 37 38 39 | |
aiperf.common.messages.progress_messages¶
ProcessingStatsMessage
¶
Bases: BaseServiceMessage
Message for processing stats. Sent by the records manager to the system controller to report the stats of the profile run.
Source code in aiperf/common/messages/progress_messages.py
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
ProfileProgressMessage
¶
Bases: BaseServiceMessage
Message for profile progress. Sent by the timing manager to the system controller to report the progress of the profile run.
Source code in aiperf/common/messages/progress_messages.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | |
ProfileResultsMessage
¶
Bases: BaseServiceMessage
Message for profile results.
Source code in aiperf/common/messages/progress_messages.py
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | |
RecordsProcessingStatsMessage
¶
Bases: BaseServiceMessage
Message for processing stats. Sent by the RecordsManager to report the stats of the profile run. This contains the stats for a single credit phase only.
Source code in aiperf/common/messages/progress_messages.py
74 75 76 77 78 79 80 81 82 83 84 85 86 87 | |
SweepProgressMessage
¶
Bases: BaseServiceMessage
Message for sweep progress.
Source code in aiperf/common/messages/progress_messages.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 | |
aiperf.common.messages.service_messages¶
BaseServiceErrorMessage
¶
Bases: BaseServiceMessage
Base message containing error data.
Source code in aiperf/common/messages/service_messages.py
97 98 99 100 101 102 | |
BaseServiceMessage
¶
Bases: Message
Base message that is sent from a service. Requires a service_id field to specify the service that sent the message.
Source code in aiperf/common/messages/service_messages.py
24 25 26 27 28 29 30 31 | |
BaseStatusMessage
¶
Bases: BaseServiceMessage
Base message containing status data. This message is sent by a service to the system controller to report its status.
Source code in aiperf/common/messages/service_messages.py
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | |
HeartbeatMessage
¶
Bases: BaseStatusMessage
Message containing heartbeat data. This message is sent by a service to the system controller to indicate that it is still running.
Source code in aiperf/common/messages/service_messages.py
72 73 74 75 76 77 78 | |
NotificationMessage
¶
Bases: BaseServiceMessage
Message containing a notification from a service. This is used to notify other services of events.
Source code in aiperf/common/messages/service_messages.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 | |
RegistrationMessage
¶
Bases: BaseStatusMessage
Message containing registration data. This message is sent by a service to the system controller to register itself.
Source code in aiperf/common/messages/service_messages.py
62 63 64 65 66 67 68 69 | |
StatusMessage
¶
Bases: BaseStatusMessage
Message containing status data. This message is sent by a service to the system controller to report its status.
Source code in aiperf/common/messages/service_messages.py
54 55 56 57 58 59 | |
aiperf.common.mixins.aiperf_lifecycle_mixin¶
AIPerfLifecycleMixin
¶
Bases: HooksMixin, AsyncTaskManagerMixin, AIPerfLoggerMixin
Mixin to add task support to a class. It abstracts away the details of the
:class:AIPerfTask and provides a simple interface for registering and running tasks.
It hooks into the :meth:HooksMixin.on_start and :meth:HooksMixin.on_stop hooks to
start and stop the tasks.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 | |
is_initialized()
¶
Check if the lifecycle has been initialized.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
43 44 45 | |
run_and_wait_for_start()
async
¶
Start the lifecycle in the background and wait until the lifecycle is initialized and started.
Will call the :meth:HooksMixin.on_init hooks, followed by the :meth:HooksMixin.on_start hooks.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
90 91 92 93 94 95 96 97 98 | |
run_async()
async
¶
Start the lifecycle in the background. Will call the :meth:HooksMixin.on_init hooks,
followed by the :meth:HooksMixin.on_start hooks. Will return immediately.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
83 84 85 86 87 88 | |
shutdown()
async
¶
Shutdown the lifecycle. Will call the :meth:HooksMixin.on_stop hooks,
followed by the :meth:HooksMixin.on_cleanup hooks.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
115 116 117 118 | |
wait_for_initialize()
async
¶
Wait for the lifecycle to be initialized. Will wait until the :meth:HooksMixin.on_init hooks have been called.
Will return immediately if the lifecycle is already initialized.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
100 101 102 103 | |
wait_for_shutdown()
async
¶
Wait for the lifecycle to be shutdown. Will wait until the :meth:HooksMixin.on_stop hooks have been called.
Will return immediately if the lifecycle is already shutdown.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
110 111 112 113 | |
wait_for_start()
async
¶
Wait for the lifecycle to be started. Will wait until the :meth:HooksMixin.on_start hooks have been called.
Will return immediately if the lifecycle is already started.
Source code in aiperf/common/mixins/aiperf_lifecycle_mixin.py
105 106 107 108 | |
aiperf.common.mixins.aiperf_logger_mixin¶
AIPerfLoggerMixin
¶
Bases: BaseMixin
Mixin to provide lazy evaluated logging for f-strings.
This mixin provides a logger with lazy evaluation support for f-strings, and direct log functions for all standard and custom logging levels.
see :class:AIPerfLogger for more details.
Usage
class MyClass(AIPerfLoggerMixin): def init(self): super().init() self.trace(lambda: f"Processing {item} of {count} ({item / count * 100}% complete)") self.info("Simple string message") self.debug(lambda i=i: f"Binding loop variable: {i}") self.warning("Warning message: %s", "legacy support") self.success("Benchmark completed successfully") self.notice("Warmup has completed") self.exception(f"Direct f-string usage: {e}")
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | |
critical(message, *args, **kwargs)
¶
Log a critical message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
96 97 98 99 | |
debug(message, *args, **kwargs)
¶
Log a debug message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
61 62 63 64 | |
error(message, *args, **kwargs)
¶
Log an error message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
86 87 88 89 | |
exception(message, *args, **kwargs)
¶
Log an exception message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
91 92 93 94 | |
info(message, *args, **kwargs)
¶
Log an info message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
66 67 68 69 | |
log(level, message, *args, **kwargs)
¶
Log a message at a specified level with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
49 50 51 52 53 54 | |
notice(message, *args, **kwargs)
¶
Log a notice message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
71 72 73 74 | |
success(message, *args, **kwargs)
¶
Log a success message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
81 82 83 84 | |
trace(message, *args, **kwargs)
¶
Log a trace message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
56 57 58 59 | |
warning(message, *args, **kwargs)
¶
Log a warning message with lazy evaluation.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
76 77 78 79 | |
AIPerfLoggerProtocol
¶
Bases: Protocol
Protocol to provide lazy evaluated logging for f-strings.
Source code in aiperf/common/mixins/aiperf_logger_mixin.py
109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 | |
aiperf.common.mixins.aiperf_profile_mixin¶
AIPerfProfileMixin
¶
Bases: HooksMixin
Mixin to add profile-related hook support to a class.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | |
configure_profile(message)
async
¶
Configure the profile.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
27 28 29 30 | |
run_profile()
async
¶
Run the profile.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | |
stop_profile()
async
¶
Request the profile to stop.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
58 59 60 | |
wait_for_profile_configured()
async
¶
Wait for the profile to be configured.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
62 63 64 | |
wait_for_profile_started()
async
¶
Wait for the profile to start.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
66 67 68 | |
wait_for_profile_stopped()
async
¶
Wait for the profile to stop.
Source code in aiperf/common/mixins/aiperf_profile_mixin.py
70 71 72 | |
aiperf.common.mixins.aiperf_task_mixin¶
AIPerfTaskMixin
¶
Bases: HooksMixin, AsyncTaskManagerMixin
Mixin to add aiperf_task support to a class.
It hooks into the :meth:HooksMixin.on_init and :meth:HooksMixin.on_stop hooks to
start and stop the tasks.
Source code in aiperf/common/mixins/aiperf_task_mixin.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | |
initialize()
async
¶
Initialize the task.
Source code in aiperf/common/mixins/aiperf_task_mixin.py
38 39 40 | |
start()
async
¶
Start the task.
Source code in aiperf/common/mixins/aiperf_task_mixin.py
42 43 44 | |
stop()
async
¶
Stop the task.
Source code in aiperf/common/mixins/aiperf_task_mixin.py
46 47 48 | |
aiperf.common.mixins.async_task_manager_mixin¶
AsyncTaskManagerMixin
¶
Bases: BaseMixin
Mixin to manage a set of async tasks.
Source code in aiperf/common/mixins/async_task_manager_mixin.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | |
cancel_all_tasks(timeout=TASK_CANCEL_TIMEOUT_SHORT)
async
¶
Cancel all tasks in the set and wait for up to timeout seconds for them to complete.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout
|
float
|
The timeout to wait for the tasks to complete. |
TASK_CANCEL_TIMEOUT_SHORT
|
Source code in aiperf/common/mixins/async_task_manager_mixin.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | |
execute_async(coro)
¶
Create a task from a coroutine and add it to the set of tasks, and return immediately. The task will be automatically cleaned up when it completes.
Source code in aiperf/common/mixins/async_task_manager_mixin.py
19 20 21 22 23 24 25 26 | |
wait_for_tasks()
async
¶
Wait for all current tasks to complete.
Source code in aiperf/common/mixins/async_task_manager_mixin.py
28 29 30 | |
AsyncTaskManagerProtocol
¶
Bases: Protocol
Protocol to manage a set of async tasks.
Source code in aiperf/common/mixins/async_task_manager_mixin.py
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | |
cancel_all_tasks(timeout=TASK_CANCEL_TIMEOUT_SHORT)
async
¶
Cancel all tasks in the set and wait for up to timeout seconds for them to complete.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout
|
float
|
The timeout to wait for the tasks to complete. |
TASK_CANCEL_TIMEOUT_SHORT
|
Source code in aiperf/common/mixins/async_task_manager_mixin.py
68 69 70 71 72 73 74 75 | |
execute_async(coro)
¶
Create a task from a coroutine and add it to the set of tasks, and return immediately. The task will be automatically cleaned up when it completes.
Source code in aiperf/common/mixins/async_task_manager_mixin.py
59 60 61 62 63 | |
stop()
async
¶
Stop all tasks in the set and wait for them to complete.
Source code in aiperf/common/mixins/async_task_manager_mixin.py
65 66 | |
aiperf.common.mixins.base_mixin¶
BaseMixin
¶
Base mixin class.
This Mixin creates a contract that Mixins should always pass **kwargs to super().init, regardless of whether they extend another mixin or not.
This will ensure that the BaseMixin is the last mixin to have its init method called, which means that all other mixins will have a proper chain of init methods with the correct arguments and no accidental broken inheritance.
Source code in aiperf/common/mixins/base_mixin.py
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
aiperf.common.mixins.hooks_mixin¶
HooksMixin
¶
Bases: BaseMixin
Mixin to add hook support to a class. It abstracts away the details of the
:class:HookSystem and provides a simple interface for registering and running hooks.
Source code in aiperf/common/mixins/hooks_mixin.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | |
__init__(**kwargs)
¶
Initialize the hook system and register all functions that are decorated with a hook decorator.
Source code in aiperf/common/mixins/hooks_mixin.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | |
get_hooks(hook_type)
¶
Get all the registered hooks for the given hook type. See :meth:HookSystem.get_hooks.
Source code in aiperf/common/mixins/hooks_mixin.py
62 63 64 | |
register_hook(hook_type, func)
¶
Register a hook function for a given hook type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
hook_type
|
HookType
|
The hook type to register the function for. |
required |
func
|
Callable
|
The function to register. |
required |
Source code in aiperf/common/mixins/hooks_mixin.py
45 46 47 48 49 50 51 52 | |
run_hooks(hook_type, *args, **kwargs)
async
¶
Run all the hooks serially. See :meth:HookSystem.run_hooks.
Source code in aiperf/common/mixins/hooks_mixin.py
54 55 56 | |
run_hooks_async(hook_type, *args, **kwargs)
async
¶
Run all the hooks concurrently. See :meth:HookSystem.run_hooks_async.
Source code in aiperf/common/mixins/hooks_mixin.py
58 59 60 | |
aiperf.common.mixins.process_health_mixin¶
ProcessHealthMixin
¶
Bases: BaseMixin
Mixin to provide process health information.
Source code in aiperf/common/mixins/process_health_mixin.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
get_process_health()
¶
Get the process health information for the current process.
Source code in aiperf/common/mixins/process_health_mixin.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
aiperf.common.models.base_models¶
AIPerfBaseModel
¶
Bases: BaseModel
Base model for all AIPerf Pydantic models. This class is configured to allow arbitrary types to be used as fields as to allow for more flexible model definitions by end users without breaking the existing code.
Source code in aiperf/common/models/base_models.py
10 11 12 13 14 15 16 | |
ExcludeIfNoneMixin
¶
Bases: AIPerfBaseModel
Mixin to exclude fields from the serialized model if they are None.
The @exclude_if_none decorator can be used to specify which fields should be excluded from the serialized model if they are None.
Source code in aiperf/common/models/base_models.py
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | |
exclude_if_none(field_names)
¶
Decorator to set the _exclude_if_none_fields class attribute to the set of field names that should be excluded if they are None.
Source code in aiperf/common/models/base_models.py
19 20 21 22 23 24 25 26 27 28 29 30 | |
aiperf.common.models.credit_models¶
CreditPhaseConfig
¶
Bases: AIPerfBaseModel
Model for phase credit config. This is used by the TimingManager to configure the credit phases.
Source code in aiperf/common/models/credit_models.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
is_valid
property
¶
A phase config is valid if it is exactly one of the following: - is_time_based (expected_duration_sec is set and > 0) - is_request_count_based (total_expected_requests is set and > 0)
CreditPhaseStats
¶
Bases: CreditPhaseConfig
Model for phase credit stats. Extends the CreditPhaseConfig fields to track the progress of the credit phases. How many credits were dropped and how many were returned, as well as the progress percentage of the phase.
Source code in aiperf/common/models/credit_models.py
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | |
in_flight
property
¶
Calculate the number of in-flight credits (sent but not completed).
should_send
property
¶
Whether the phase should send more credits.
from_phase_config(phase_config)
classmethod
¶
Create a CreditPhaseStats from a CreditPhaseConfig. This is used to initialize the stats for a phase.
Source code in aiperf/common/models/credit_models.py
125 126 127 128 129 130 131 132 | |
PhaseProcessingStats
¶
Bases: AIPerfBaseModel
Model for phase processing stats. How many requests were processed and how many errors were encountered.
Source code in aiperf/common/models/credit_models.py
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | |
total_records
property
¶
The total number of records processed successfully or in error.
aiperf.common.models.dataset_models¶
Conversation
¶
Bases: AIPerfBaseModel
A dataset representation of a full conversation.
A conversation is a sequence of turns between a user and an endpoint, and it contains the session ID and all the turns that consists the conversation.
Source code in aiperf/common/models/dataset_models.py
63 64 65 66 67 68 69 70 71 72 73 | |
Turn
¶
Bases: AIPerfBaseModel
A dataset representation of a single turn within a conversation.
A turn is a single interaction between a user and an AI assistant, and it contains timestamp, delay, and raw data that user sends in each turn.
Source code in aiperf/common/models/dataset_models.py
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | |
aiperf.common.models.error_models¶
ErrorDetails
¶
Bases: AIPerfBaseModel
Encapsulates details about an error.
Source code in aiperf/common/models/error_models.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | |
__eq__(other)
¶
Check if the error details are equal by comparing the code, type, and message.
Source code in aiperf/common/models/error_models.py
26 27 28 29 30 31 32 33 34 | |
__hash__()
¶
Hash the error details by hashing the code, type, and message.
Source code in aiperf/common/models/error_models.py
36 37 38 | |
from_exception(e)
classmethod
¶
Create an error details object from an exception.
Source code in aiperf/common/models/error_models.py
40 41 42 43 44 45 46 | |
ErrorDetailsCount
¶
Bases: AIPerfBaseModel
Count of error details.
Source code in aiperf/common/models/error_models.py
49 50 51 52 53 54 55 56 | |
aiperf.common.models.health_models¶
ProcessHealth
¶
Bases: AIPerfBaseModel
Model for process health data.
Source code in aiperf/common/models/health_models.py
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | |
aiperf.common.models.record_models¶
InferenceServerResponse
¶
Bases: AIPerfBaseModel
Response from a inference client.
Source code in aiperf/common/models/record_models.py
52 53 54 55 56 57 58 | |
MetricResult
¶
Bases: AIPerfBaseModel
The result values of a single metric.
Source code in aiperf/common/models/record_models.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | |
ParsedResponseRecord
¶
Bases: AIPerfBaseModel
Record of a request and its associated responses, already parsed and ready for metrics.
Source code in aiperf/common/models/record_models.py
290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 | |
end_perf_ns
cached
property
¶
Get the end time of the request in nanoseconds (perf_counter_ns). If request.end_perf_ns is not set, use the time of the last response. If there are no responses, use sys.maxsize.
has_error
cached
property
¶
Check if the response record has an error.
request_duration_ns
cached
property
¶
Get the duration of the request in nanoseconds.
start_perf_ns
cached
property
¶
Get the start time of the request in nanoseconds (perf_counter_ns).
timestamp_ns
cached
property
¶
Get the wall clock timestamp of the request in nanoseconds. DO NOT USE FOR LATENCY CALCULATIONS. (time.time_ns).
tokens_per_second
cached
property
¶
Get the number of tokens per second of the request.
valid
cached
property
¶
Check if the response record is valid.
Checks: - Request has no errors - Has at least one response - Start time is before the end time - Response timestamps are within valid ranges
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the record is valid, False otherwise. |
RequestRecord
¶
Bases: AIPerfBaseModel
Record of a request with its associated responses.
Source code in aiperf/common/models/record_models.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | |
delayed
property
¶
Check if the request was delayed.
has_error
property
¶
Check if the request record has an error.
inter_token_latency_ns
property
¶
Get the interval between responses in nanoseconds.
time_to_first_response_ns
property
¶
Get the time to the first response in nanoseconds.
time_to_last_response_ns
property
¶
Get the time to the last response in nanoseconds.
time_to_second_response_ns
property
¶
Get the time to the second response in nanoseconds.
valid
property
¶
Check if the request record is valid by ensuring that the start time and response timestamps are within valid ranges.
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True if the record is valid, False otherwise. |
token_latency_ns(index)
¶
Get the latency of a token in nanoseconds.
Source code in aiperf/common/models/record_models.py
256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | |
ResponseData
¶
Bases: AIPerfBaseModel
Base class for all response data.
Source code in aiperf/common/models/record_models.py
273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 | |
SSEField
¶
Bases: AIPerfBaseModel
Base model for a single field in an SSE message.
Source code in aiperf/common/models/record_models.py
74 75 76 77 78 79 80 81 82 83 84 | |
SSEMessage
¶
Bases: InferenceServerResponse
Individual SSE message from an SSE stream. Delimited by
.
Source code in aiperf/common/models/record_models.py
87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | |
extract_data_content()
¶
Extract the data contents from the SSE message as a list of strings. Note that the SSE spec specifies that each data content should be combined and delimited by a single . We have left it as a list to allow the caller to decide how to handle the data.
Returns:
list[str]: A list of strings containing the data contents of the SSE message.
Source code in aiperf/common/models/record_models.py
96 97 98 99 100 101 102 103 104 105 106 107 108 | |
TextResponse
¶
Bases: InferenceServerResponse
Raw text response from a inference client including an optional content type.
Source code in aiperf/common/models/record_models.py
61 62 63 64 65 66 67 68 69 70 71 | |
aiperf.common.models.service_models¶
ServiceRunInfo
¶
Bases: BaseModel
Base model for tracking service run information.
Source code in aiperf/common/models/service_models.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | |
aiperf.common.models.worker_models¶
WorkerPhaseTaskStats
¶
Bases: AIPerfBaseModel
Stats for the tasks that have been sent to the worker for a given credit phase.
Source code in aiperf/common/models/worker_models.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | |
in_progress
property
¶
The number of tasks that are currently in progress.
This is the total number of tasks sent to the worker minus the number of failed and successfully completed tasks.
aiperf.common.service.base_component_service¶
BaseComponentService
¶
Bases: BaseService
Base class for all Component services.
This class provides a common interface for all Component services in the AIPerf framework such as the Timing Manager, Dataset Manager, etc.
It extends the BaseService by: - Subscribing to the command message_type - Processing command messages - Sending registration requests to the system controller - Sending heartbeat notifications to the system controller - Sending status notifications to the system controller - Helpers to create heartbeat, registration, and status messages - Request the appropriate communication clients for a component service
Source code in aiperf/common/service/base_component_service.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 | |
create_heartbeat_message()
¶
Create a heartbeat notification message.
Source code in aiperf/common/service/base_component_service.py
219 220 221 222 223 224 225 | |
create_registration_message()
¶
Create a registration request message.
Source code in aiperf/common/service/base_component_service.py
227 228 229 230 231 232 | |
create_status_message(state)
¶
Create a status notification message.
Source code in aiperf/common/service/base_component_service.py
234 235 236 237 238 239 240 | |
process_command_message(message)
async
¶
Process a command message received from the controller.
This method will process the command message and execute the appropriate action.
Source code in aiperf/common/service/base_component_service.py
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 | |
register()
async
¶
Publish a registration request to the system controller.
This method should be called after the service has been initialized and is ready to start processing messages.
Source code in aiperf/common/service/base_component_service.py
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | |
register_command_callback(cmd, callback)
¶
Register a single callback for a command.
Source code in aiperf/common/service/base_component_service.py
195 196 197 198 199 200 201 | |
send_heartbeat()
async
¶
Send a heartbeat notification to the system controller.
Source code in aiperf/common/service/base_component_service.py
106 107 108 109 110 111 112 113 114 115 | |
aiperf.common.service.base_controller_service¶
BaseControllerService
¶
Bases: BaseService
Base class for all controller services, such as the System Controller.
This class provides a common interface for all controller services in the AIPerf framework. It inherits from the BaseService class and implements the required methods for controller services.
It extends the BaseService by: - Starting the service automatically when the run hook is called - Helpers to create command messages to be sent to a specific service - Request the appropriate communication clients for a controller service
Source code in aiperf/common/service/base_controller_service.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
create_command_message(command, target_service_id, target_service_type=None, data=None)
¶
Create a command message to be sent to a specific service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
command
|
CommandType
|
The command to send |
required |
target_service_id
|
str | None
|
The ID of the service to send the command to |
required |
target_service_type
|
ServiceType | None
|
The type of the service to send the command to |
None
|
data
|
BaseModel | None
|
Optional data to send with the command. |
None
|
Returns:
| Type | Description |
|---|---|
CommandMessage
|
A command message |
Source code in aiperf/common/service/base_controller_service.py
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
aiperf.common.service.base_service¶
BaseService
¶
Bases: BaseServiceInterface, ABC, AIPerfTaskMixin, AIPerfLoggerMixin
Base class for all AIPerf services, providing common functionality for communication, state management, and lifecycle operations.
This class provides the foundation for implementing the various services of the AIPerf system. Some of the abstract methods are implemented here, while others are still required to be implemented by derived classes.
Source code in aiperf/common/service/base_service.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 | |
is_initialized
property
¶
Check if service is initialized.
Returns:
| Type | Description |
|---|---|
bool
|
True if service is initialized, False otherwise |
state
property
¶
The current state of the service.
configure(message)
async
¶
Configure the service with the given configuration. This method implements
the BaseServiceInterface.configure method.
This method will: - Call all registered AIPerfHook.ON_CONFIGURE hooks
Source code in aiperf/common/service/base_service.py
326 327 328 329 330 331 332 333 | |
initialize()
async
¶
Initialize the service communication and signal handlers. This method implements
the BaseServiceInterface.initialize method.
This method will:
- Set the service to ServiceState.INITIALIZING state
- Initialize communication
- Call all registered AIPerfHook.ON_INIT hooks
- Set the service to ServiceState.READY state
- Set the initialized asyncio event
Source code in aiperf/common/service/base_service.py
137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
run_forever()
async
¶
Run the service in a loop until the stop event is set. This method implements
the BaseServiceInterface.run_forever method.
This method will:
- Call the initialize method to initialize the service
- Call all registered AIPerfHook.RUN hooks
- Wait for the stop event to be set
- Shuts down the service when the stop event is set
This method will be called as the main entry point for the service.
Source code in aiperf/common/service/base_service.py
158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 | |
set_state(state)
async
¶
Set the state of the service. This method implements
the BaseServiceInterface.set_state method.
This method will:
- Set the service state to the given state
- Call all registered AIPerfHook.ON_SET_STATE hooks
Source code in aiperf/common/service/base_service.py
126 127 128 129 130 131 132 133 134 135 | |
start()
async
¶
Start the service and its components. This method implements
the BaseServiceInterface.start method.
This method should be called to start the service after it has been initialized and configured.
This method will:
- Set the service to ServiceState.STARTING state
- Call all registered AIPerfHook.ON_START hooks
- Set the service to ServiceState.RUNNING state
Source code in aiperf/common/service/base_service.py
231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 | |
stop()
async
¶
Stop the service and clean up its components. This method implements
the BaseServiceInterface.stop method.
This method will:
- Set the service to ServiceState.STOPPING state
- Call all registered AIPerfHook.ON_STOP hooks
- Shutdown the service communication component
- Call all registered AIPerfHook.ON_CLEANUP hooks
- Set the service to ServiceState.STOPPED state
Source code in aiperf/common/service/base_service.py
261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 | |
aiperf.common.service.base_service_interface¶
BaseServiceInterface
¶
Bases: ABC
Base interface for all services.
This class provides the base foundation for which every service should provide. Some methods are required to be implemented by derived classes, while others are meant to be implemented by the base class.
Source code in aiperf/common/service/base_service_interface.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
service_type
abstractmethod
property
¶
The type/name of the service.
This property should be implemented by derived classes to specify the type/name of the service.
configure(message)
abstractmethod
async
¶
Configure the service with the given configuration.
This method will be implemented by the base class, and extra
functionality can be added by derived classes via the @on_configure
decorator.
Source code in aiperf/common/service/base_service_interface.py
66 67 68 69 70 71 72 73 74 | |
initialize()
abstractmethod
async
¶
Initialize the service.
This method will be implemented by the base class.
Source code in aiperf/common/service/base_service_interface.py
37 38 39 40 41 42 43 | |
run_forever()
abstractmethod
async
¶
Run the service. This method will be the primary entry point for the service and will be called by the bootstrap script. It should not return until the service is completely shutdown.
This method will be implemented by the base class. Any additional
functionality can be added by derived classes via the @on_run
decorator.
Source code in aiperf/common/service/base_service_interface.py
76 77 78 79 80 81 82 83 84 85 86 | |
set_state(state)
abstractmethod
async
¶
Set the state of the service.
This method will be implemented by the base class, and extra
functionality can be added by derived classes via the @on_set_state
decorator.
Source code in aiperf/common/service/base_service_interface.py
27 28 29 30 31 32 33 34 35 | |
start()
abstractmethod
async
¶
Start the service. It should be called after the service has been initialized and configured.
This method will be implemented by the base class, and extra
functionality can be added by derived classes via the @on_start
decorator.
Source code in aiperf/common/service/base_service_interface.py
45 46 47 48 49 50 51 52 53 54 | |
stop()
abstractmethod
async
¶
Stop the service.
This method will be implemented by the base class, and extra
functionality can be added by derived classes via the @on_stop
decorator.
Source code in aiperf/common/service/base_service_interface.py
56 57 58 59 60 61 62 63 64 | |
aiperf.common.tokenizer¶
Tokenizer
¶
This class provides a simplified interface for using Huggingface tokenizers, with default arguments for common operations.
Source code in aiperf/common/tokenizer.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | |
bos_token_id
property
¶
Return the beginning-of-sequence (BOS) token ID.
__call__(text, **kwargs)
¶
Call the underlying Huggingface tokenizer with default arguments, which can be overridden by kwargs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
The input text to tokenize. |
required |
Returns:
| Type | Description |
|---|---|
BatchEncoding
|
A BatchEncoding object containing the tokenized output. |
Source code in aiperf/common/tokenizer.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 | |
__init__()
¶
Initialize the tokenizer with default values for call, encode, and decode.
Source code in aiperf/common/tokenizer.py
28 29 30 31 32 33 34 35 | |
__repr__()
¶
Return a string representation of the underlying tokenizer.
Returns:
| Type | Description |
|---|---|
str
|
The string representation of the tokenizer. |
Source code in aiperf/common/tokenizer.py
119 120 121 122 123 124 125 126 | |
__str__()
¶
Return a user-friendly string representation of the underlying tokenizer.
Returns:
| Type | Description |
|---|---|
str
|
The string representation of the tokenizer. |
Source code in aiperf/common/tokenizer.py
128 129 130 131 132 133 134 135 | |
decode(token_ids, **kwargs)
¶
Decode a list of token IDs back into a string.
This method calls the underlying Huggingface tokenizer's decode method with default arguments, which can be overridden by kwargs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
token_ids
|
A list of token IDs to decode. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The decoded string. |
Source code in aiperf/common/tokenizer.py
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | |
encode(text, **kwargs)
¶
Encode the input text into a list of token IDs.
This method calls the underlying Huggingface tokenizer's encode method with default arguments, which can be overridden by kwargs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
The input text to encode. |
required |
Returns:
| Type | Description |
|---|---|
list[int]
|
A list of token IDs. |
Source code in aiperf/common/tokenizer.py
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 | |
from_pretrained(name, trust_remote_code=False, revision='main')
classmethod
¶
Factory to load a tokenizer for the given pretrained model name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The name or path of the pretrained tokenizer model. |
required |
trust_remote_code
|
bool
|
Whether to trust remote code when loading the tokenizer. |
False
|
revision
|
str
|
The specific model version to use. |
'main'
|
Source code in aiperf/common/tokenizer.py
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | |
aiperf.common.types¶
aiperf.common.utils¶
call_all_functions(funcs, *args, **kwargs)
async
¶
Call all functions in the list with the given name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obj
|
The object to call the functions on. |
required | |
func_names
|
The names of the functions to call. |
required | |
*args
|
The arguments to pass to the functions. |
()
|
|
**kwargs
|
The keyword arguments to pass to the functions. |
{}
|
Raises:
| Type | Description |
|---|---|
AIPerfMultiError
|
If any of the functions raise an exception. |
Source code in aiperf/common/utils.py
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | |
call_all_functions_self(self_, funcs, *args, **kwargs)
async
¶
Call all functions in the list with the given name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obj
|
The object to call the functions on. |
required | |
func_names
|
The names of the functions to call. |
required | |
*args
|
The arguments to pass to the functions. |
()
|
|
**kwargs
|
The keyword arguments to pass to the functions. |
{}
|
Raises:
| Type | Description |
|---|---|
AIPerfMultiError
|
If any of the functions raise an exception. |
Source code in aiperf/common/utils.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
load_json_str(json_str, func=lambda x: x)
¶
Deserializes JSON encoded string into Python object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
-
|
json_str
|
string JSON encoded string |
required |
-
|
func
|
callable A function that takes deserialized JSON object. This can be used to run validation checks on the object. Defaults to identity function. |
required |
Source code in aiperf/common/utils.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | |
yield_to_event_loop()
async
¶
Yield to the event loop. This forces the current coroutine to yield and allow other coroutines to run, preventing starvation. Use this when you do not want to delay your coroutine via sleep, but still want to allow other coroutines to run if there is a potential for an infinite loop.
Source code in aiperf/common/utils.py
101 102 103 104 105 106 107 | |
aiperf.data_exporter.console_error_exporter¶
ConsoleErrorExporter
¶
A class that exports error data to the console
Source code in aiperf/data_exporter/console_error_exporter.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | |
aiperf.data_exporter.console_exporter¶
ConsoleExporter
¶
A class that exports data to the console
Source code in aiperf/data_exporter/console_exporter.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | |
aiperf.data_exporter.exporter_config¶
aiperf.data_exporter.exporter_manager¶
ExporterManager
¶
ExporterManager is responsible for exporting records using all registered data exporters.
Source code in aiperf/data_exporter/exporter_manager.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | |
aiperf.data_exporter.json_exporter¶
JsonExportData
¶
Bases: BaseModel
Data to be exported to a JSON file.
Source code in aiperf/data_exporter/json_exporter.py
18 19 20 21 22 23 24 25 26 | |
JsonExporter
¶
A class to export records to a JSON file.
Source code in aiperf/data_exporter/json_exporter.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | |
aiperf.progress.progress_models¶
BenchmarkSuiteCompletionTrigger
¶
Bases: CaseInsensitiveStrEnum
Determines how the suite completion is determined in order to know how to track the progress.
Source code in aiperf/progress/progress_models.py
101 102 103 104 105 106 107 108 | |
BenchmarkSuiteProgress
¶
Bases: BaseModel, ABC
State of the suite progress.
Source code in aiperf/progress/progress_models.py
266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 | |
BenchmarkSuiteType
¶
Bases: CaseInsensitiveStrEnum
Determines the type of suite to know how to track the progress.
Source code in aiperf/progress/progress_models.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | |
CUSTOM = 'custom'
class-attribute
instance-attribute
¶
User defined suite type. TBD
MULTI_PROFILE = 'multi_profile'
class-attribute
instance-attribute
¶
An suite with multiple profile runs. As opposed to a sweep, more than one parameter can be varied. TBD
MULTI_SWEEP = 'multi_sweep'
class-attribute
instance-attribute
¶
An suite with multiple sweep runs over multiple varying parameters. TBD
SINGLE_PROFILE = 'single_profile'
class-attribute
instance-attribute
¶
An suite with a single profile run.
SINGLE_SWEEP = 'single_sweep'
class-attribute
instance-attribute
¶
An suite with a single sweep over one or more varying parameters. TBD
ProfileCompletionTrigger
¶
Bases: CaseInsensitiveStrEnum
Determines how the profile completion is determined in order to know how to track the progress.
Source code in aiperf/progress/progress_models.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | |
CUSTOM = 'custom'
class-attribute
instance-attribute
¶
User defined trigger. TBD
GOODPUT_THRESHOLD = 'goodput_threshold'
class-attribute
instance-attribute
¶
The profile will run until the goodput threshold is met. TDB
REQUEST_COUNT = 'request_count'
class-attribute
instance-attribute
¶
The profile will run for a fixed number of requests.
STABILIZATION_BASED = 'stabilization_based'
class-attribute
instance-attribute
¶
The profile will run until the metrics stabilize. TDB
TIME_BASED = 'time_based'
class-attribute
instance-attribute
¶
The profile will run for a fixed amount of time.
ProfileProgress
¶
Bases: BaseModel
State of the profile progress.
Source code in aiperf/progress/progress_models.py
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 | |
ProfileSuiteProgress
¶
Bases: BenchmarkSuiteProgress
State of a profile based suite with 1 or more profile runs.
Source code in aiperf/progress/progress_models.py
316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 | |
SweepCompletionTrigger
¶
Bases: CaseInsensitiveStrEnum
Determines how the sweep completion is determined in order to know how to track the progress.
Source code in aiperf/progress/progress_models.py
33 34 35 36 37 38 39 40 41 42 43 44 45 46 | |
COMPLETED_PROFILES = 'completed_profiles'
class-attribute
instance-attribute
¶
The sweep will run until all profiles are completed.
CUSTOM = 'custom'
class-attribute
instance-attribute
¶
User defined trigger. TBD
GOODPUT_THRESHOLD = 'goodput_threshold'
class-attribute
instance-attribute
¶
The sweep will run until the goodput threshold is met. TDB
STABILIZATION_BASED = 'stabilization_based'
class-attribute
instance-attribute
¶
The sweep will run until the metrics stabilize. TDB
SweepMultiParamOrder
¶
Bases: CaseInsensitiveStrEnum
Determines the order in which the sweep parameters are tested for a multi-parameter sweep. This is only applicable for multi-parameter sweeps.
Source code in aiperf/progress/progress_models.py
84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | |
BREADTH_FIRST = 'breadth_first'
class-attribute
instance-attribute
¶
The parameters are tested in breadth-first order.
CUSTOM = 'custom'
class-attribute
instance-attribute
¶
User defined order. TBD
DEPTH_FIRST = 'depth_first'
class-attribute
instance-attribute
¶
The parameters are tested in depth-first order.
RANDOM = 'random'
class-attribute
instance-attribute
¶
The parameters are tested in random order. TBD
SweepParamOrder
¶
Bases: CaseInsensitiveStrEnum
Determines the order in which the sweep parameters are tested.
Source code in aiperf/progress/progress_models.py
68 69 70 71 72 73 74 75 76 77 78 79 80 81 | |
ASCENDING = 'ascending'
class-attribute
instance-attribute
¶
The parameters are tested in ascending order.
CUSTOM = 'custom'
class-attribute
instance-attribute
¶
User defined order. TBD
DESCENDING = 'descending'
class-attribute
instance-attribute
¶
The parameters are tested in descending order.
RANDOM = 'random'
class-attribute
instance-attribute
¶
The parameters are tested in random order. TBD
SweepParamType
¶
Bases: CaseInsensitiveStrEnum
Determines the type of sweep parameter.
Source code in aiperf/progress/progress_models.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | |
BOOLEAN = 'boolean'
class-attribute
instance-attribute
¶
The parameter is a boolean.
CUSTOM = 'custom'
class-attribute
instance-attribute
¶
User defined parameter type. TBD
FLOAT = 'float'
class-attribute
instance-attribute
¶
The parameter is a float.
INT = 'int'
class-attribute
instance-attribute
¶
The parameter is an integer.
STRING = 'string'
class-attribute
instance-attribute
¶
The parameter is a string.
SweepProgress
¶
Bases: BaseModel
State of the sweep progress.
Source code in aiperf/progress/progress_models.py
217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 | |
SweepSuiteProgress
¶
Bases: BenchmarkSuiteProgress
State of a sweep based suite with 1 or more sweep runs.
Source code in aiperf/progress/progress_models.py
343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 | |
next_profile()
¶
Get the next profile to run.
Returns:
| Type | Description |
|---|---|
ProfileProgress | None
|
The next profile to run, or None if there are no more profiles to run. |
Source code in aiperf/progress/progress_models.py
358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 | |
next_sweep()
¶
Get the next sweep to run.
Returns:
| Type | Description |
|---|---|
SweepProgress | None
|
The next sweep to run, or None if there are no more sweeps to run. |
Source code in aiperf/progress/progress_models.py
381 382 383 384 385 386 387 388 389 390 391 392 393 | |
aiperf.services.dataset.composer.base¶
BaseDatasetComposer
¶
Bases: ABC
Source code in aiperf/services/dataset/composer/base.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | |
create_dataset()
abstractmethod
¶
Create a set of conversation objects from the given configuration.
Returns:
| Type | Description |
|---|---|
list[Conversation]
|
list[Conversation]: A list of conversation objects. |
Source code in aiperf/services/dataset/composer/base.py
26 27 28 29 30 31 32 33 34 | |
aiperf.services.dataset.composer.custom¶
CustomDatasetComposer
¶
Bases: BaseDatasetComposer
Source code in aiperf/services/dataset/composer/custom.py
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | |
create_dataset()
¶
Create conversations from a file or directory.
Returns:
| Type | Description |
|---|---|
list[Conversation]
|
list[Conversation]: A list of conversation objects. |
Source code in aiperf/services/dataset/composer/custom.py
18 19 20 21 22 23 24 25 26 27 28 29 30 | |
aiperf.services.dataset.composer.synthetic¶
SyntheticDatasetComposer
¶
Bases: BaseDatasetComposer
Source code in aiperf/services/dataset/composer/synthetic.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | |
create_dataset()
¶
Create a synthetic conversation dataset from the given configuration.
It generates a set of conversations with a varying number of turns, where each turn contains synthetic text, image, and audio payloads.
Returns:
| Type | Description |
|---|---|
list[Conversation]
|
list[Conversation]: A list of conversation objects. |
Source code in aiperf/services/dataset/composer/synthetic.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | |
aiperf.services.dataset.dataset_manager¶
DatasetManager
¶
Bases: BaseComponentService
The DatasetManager primary responsibility is to manage the data generation or acquisition. For synthetic generation, it contains the code to generate the prompts or tokens. It will have an API for dataset acquisition of a dataset if available in a remote repository or database.
Source code in aiperf/services/dataset/dataset_manager.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 | |
service_type
property
¶
The type of service.
main()
¶
Main entry point for the dataset manager.
Source code in aiperf/services/dataset/dataset_manager.py
252 253 254 255 256 257 | |
aiperf.services.dataset.generator.audio¶
AudioGenerator
¶
Bases: BaseGenerator
A class for generating synthetic audio data.
This class provides methods to create audio samples with specified characteristics such as format (WAV, MP3), length, sampling rate, bit depth, and number of channels. It supports validation of audio parameters to ensure compatibility with chosen formats.
Source code in aiperf/services/dataset/generator/audio.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | |
generate(*args, **kwargs)
¶
Generate audio data with specified parameters.
Returns:
| Type | Description |
|---|---|
str
|
Data URI containing base64-encoded audio data with format specification |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If any of the following conditions are met: - audio length is less than 0.01 seconds - channels is not 1 (mono) or 2 (stereo) - sampling rate is not supported for MP3 format - bit depth is not supported (must be 8, 16, 24, or 32) - audio format is not supported (must be 'wav' or 'mp3') |
Source code in aiperf/services/dataset/generator/audio.py
92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | |
aiperf.services.dataset.generator.base¶
BaseGenerator
¶
Bases: ABC
Abstract base class for all data generators.
Provides a consistent interface for generating synthetic data while allowing each generator type to use its own specific configuration and runtime parameters.
Source code in aiperf/services/dataset/generator/base.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | |
generate(*args, **kwargs)
abstractmethod
¶
Generate synthetic data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*args
|
Variable length argument list (subclass-specific) |
()
|
|
**kwargs
|
Arbitrary keyword arguments (subclass-specific) |
{}
|
Returns:
| Type | Description |
|---|---|
str
|
Generated data as a string (could be text, base64 encoded media, etc.) |
Source code in aiperf/services/dataset/generator/base.py
18 19 20 21 22 23 24 25 26 27 28 29 | |
aiperf.services.dataset.generator.image¶
ImageGenerator
¶
Bases: BaseGenerator
A class that generates images from source images.
This class provides methods to create synthetic images by resizing source images (located in the 'assets/source_images' directory) to specified dimensions and converting them to a chosen image format (e.g., PNG, JPEG). The dimensions can be randomized based on mean and standard deviation values.
Source code in aiperf/services/dataset/generator/image.py
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | |
generate(*args, **kwargs)
¶
Generate an image with the configured parameters.
Returns:
| Type | Description |
|---|---|
str
|
A base64 encoded string of the generated image. |
Source code in aiperf/services/dataset/generator/image.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | |
aiperf.services.dataset.generator.prompt¶
PromptGenerator
¶
Bases: BaseGenerator
A class for generating synthetic prompts from a text corpus.
This class loads a text corpus (e.g., Shakespearean text), tokenizes it, and uses the tokenized corpus to generate synthetic prompts of specified lengths. It supports generating prompts with a target number of tokens (with optional randomization around a mean and standard deviation) and can reuse previously generated token blocks to optimize generation for certain use cases. It also allows for the creation of a pool of prefix prompts that can be randomly selected.
Source code in aiperf/services/dataset/generator/prompt.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 | |
generate(mean=None, stddev=None, hash_ids=None)
¶
Generate a synthetic prompt with the configuration parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mean
|
int | None
|
The mean of the normal distribution. |
None
|
stddev
|
int | None
|
The standard deviation of the normal distribution. |
None
|
hash_ids
|
list[int] | None
|
A list of hash indices used for token reuse. |
None
|
Returns:
| Type | Description |
|---|---|
str
|
A synthetic prompt as a string. |
Source code in aiperf/services/dataset/generator/prompt.py
100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | |
get_random_prefix_prompt()
¶
Fetch a random prefix prompt from the pool.
Returns:
| Type | Description |
|---|---|
str
|
A random prefix prompt. |
Raises:
| Type | Description |
|---|---|
InvalidStateError
|
If the prefix prompts pool is empty. |
Source code in aiperf/services/dataset/generator/prompt.py
219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 | |
aiperf.services.dataset.loader.models¶
CustomData = Annotated[SingleTurn | MooncakeTrace | MultiTurn, Field(discriminator='type')]
module-attribute
¶
A union type of all custom data types.
MooncakeTrace
¶
Bases: AIPerfBaseModel
Defines the schema for Mooncake trace data.
See https://github.com/kvcache-ai/Mooncake for more details.
Example:
{"timestamp": 1000, "input_length": 10, "output_length": 4, "hash_ids": [123, 456]}
Source code in aiperf/services/dataset/loader/models.py
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | |
MultiTurn
¶
Bases: AIPerfBaseModel
Defines the schema for multi-turn conversations.
The multi-turn custom dataset - supports multi-modal data (e.g. text, image, audio) - supports multi-turn features (e.g. delay, sessions, etc.) - supports client-side batching for each data (e.g. batch size > 1)
Source code in aiperf/services/dataset/loader/models.py
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
validate_turns_not_empty()
¶
Ensure at least one turn is provided
Source code in aiperf/services/dataset/loader/models.py
91 92 93 94 95 96 | |
SingleTurn
¶
Bases: AIPerfBaseModel
Defines the schema for single-turn data.
User can use this format to quickly provide a custom single turn dataset. Each line in the file will be treated as a single turn conversation.
The single turn type - supports multi-modal (e.g. text, image, audio) - supports client-side batching for each data (e.g. batch_size > 1) - DOES NOT support multi-turn features (e.g. delay, sessions, etc.)
Source code in aiperf/services/dataset/loader/models.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
validate_at_least_one_modality()
¶
Ensure at least one modality is provided
Source code in aiperf/services/dataset/loader/models.py
63 64 65 66 67 68 69 70 | |
validate_mutually_exclusive_fields()
¶
Ensure mutually exclusive fields are not set together
Source code in aiperf/services/dataset/loader/models.py
50 51 52 53 54 55 56 57 58 59 60 61 | |
aiperf.services.dataset.loader.mooncake_trace¶
MooncakeTraceDatasetLoader
¶
A dataset loader that loads Mooncake trace data from a file.
Loads Mooncake trace data from a file and converts the data into a list of conversations for dataset manager.
Each line in the file represents a single trace entry and will be converted to a separate conversation with a unique session ID.
Example: Fixed schedule version (Each line is a distinct session. Multi-turn is NOT supported)
{"timestamp": 1000, "input_length": 300, "output_length": 40, "hash_ids": [123, 456]}
Source code in aiperf/services/dataset/loader/mooncake_trace.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | |
convert_to_conversations(data)
¶
Convert all the Mooncake trace data to conversation objects.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, list[MooncakeTrace]]
|
A dictionary of session_id and list of Mooncake trace data. |
required |
Returns:
| Type | Description |
|---|---|
list[Conversation]
|
A list of conversations. |
Source code in aiperf/services/dataset/loader/mooncake_trace.py
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | |
load_dataset()
¶
Load Mooncake trace data from a file.
Returns:
| Type | Description |
|---|---|
dict[str, list[MooncakeTrace]]
|
A dictionary of session_id and list of Mooncake trace data. |
Source code in aiperf/services/dataset/loader/mooncake_trace.py
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | |
aiperf.services.dataset.loader.multi_turn¶
MultiTurnDatasetLoader
¶
A dataset loader that loads multi-turn data from a file.
The multi-turn type - supports multi-modal data (e.g. text, image, audio) - supports multi-turn features (e.g. delay, sessions, etc.) - supports client-side batching for each data (e.g. batch_size > 1)
NOTE: If the user specifies multiple multi-turn entries with same session ID, the loader will group them together. If the timestamps are specified, they will be sorted in ascending order later in the timing manager.
Examples: 1. Simple version
{
"session_id": "session_123",
"turns": [
{"text": "Hello", "image": "url", "delay": 0},
{"text": "Hi there", "delay": 1000}
]
}
- Batched version
{
"session_id": "session_123",
"turns": [
{"texts": ["Who are you?", "Hello world"], "images": ["/path/1.png", "/path/2.png"]},
{"texts": ["What is in the image?", "What is AI?"], "images": ["/path/3.png", "/path/4.png"]}
]
}
- Fixed schedule version
{
"session_id": "session_123",
"turns": [
{"timestamp": 0, "text": "What is deep learning?"},
{"timestamp": 1000, "text": "Who are you?"}
]
}
- Time delayed version
{
"session_id": "session_123",
"turns": [
{"delay": 0, "text": "What is deep learning?"},
{"delay": 1000, "text": "Who are you?"}
]
}
- full-featured version (multi-batch, multi-modal, multi-fielded, session-based, etc.)
{
"session_id": "session_123",
"turns": [
{
"timestamp": 1234,
"texts": [
{"name": "text_field_a", "content": ["hello", "world"]},
{"name": "text_field_b", "content": ["hi there"]}
],
"images": [
{"name": "image_field_a", "content": ["/path/1.png", "/path/2.png"]},
{"name": "image_field_b", "content": ["/path/3.png"]}
]
}
]
}
Source code in aiperf/services/dataset/loader/multi_turn.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | |
load_dataset()
¶
Load multi-turn data from a JSONL file.
Each line represents a complete multi-turn conversation with its own session_id and multiple turns.
Returns:
| Type | Description |
|---|---|
dict[str, list[MultiTurn]]
|
A dictionary mapping session_id to list of CustomData (containing the MultiTurn). |
Source code in aiperf/services/dataset/loader/multi_turn.py
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | |
aiperf.services.dataset.loader.protocol¶
aiperf.services.dataset.loader.random_pool¶
RandomPoolDatasetLoader
¶
A random pool of conversations.
A random pool of conversations is a pool of conversations that are randomly selected from a file.
Example:
{"text": "Hello world"}
Source code in aiperf/services/dataset/loader/random_pool.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | |
aiperf.services.dataset.loader.single_turn¶
SingleTurnDatasetLoader
¶
A dataset loader that loads single turn data from a file.
The single turn type - supports multi-modal data (e.g. text, image, audio) - supports client-side batching for each data (e.g. batch_size > 1) - DOES NOT support multi-turn features (e.g. delay, sessions, etc.)
Examples: 1. Single-batch, text only
{"text": "What is deep learning?"}
- Single-batch, multi-modal
{"text": "What is in the image?", "image": "/path/to/image.png"}
- Multi-batch, multi-modal
{"texts": ["Who are you?", "Hello world"], "images": ["/path/to/image.png", "/path/to/image2.png"]}
- Fixed schedule version
{"timestamp": 0, "text": "What is deep learning?"},
{"timestamp": 1000, "text": "Who are you?"},
{"timestamp": 2000, "text": "What is AI?"}
- Time delayed version
{"delay": 0, "text": "What is deep learning?"},
{"delay": 1234, "text": "Who are you?"}
- Full-featured version (Multi-batch, multi-modal, multi-fielded)
{
"texts": [
{"name": "text_field_A", "content": ["Hello", "World"]},
{"name": "text_field_B", "content": ["Hi there"]}
],
"images": [
{"name": "image_field_A", "content": ["/path/1.png", "/path/2.png"]},
{"name": "image_field_B", "content": ["/path/3.png"]}
]
}
Source code in aiperf/services/dataset/loader/single_turn.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | |
load_dataset()
¶
Load single-turn data from a JSONL file.
Each line represents a single turn conversation. Multiple turns with the same session_id (or generated UUID) are grouped together.
Returns:
| Type | Description |
|---|---|
dict[str, list[SingleTurn]]
|
A dictionary mapping session_id to list of CustomData. |
Source code in aiperf/services/dataset/loader/single_turn.py
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | |
aiperf.services.dataset.utils¶
check_file_exists(filename)
¶
Verifies that the file exists.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
The file path to verify. |
required |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If the file does not exist. |
Source code in aiperf/services/dataset/utils.py
18 19 20 21 22 23 24 25 26 27 28 | |
encode_image(img, format)
¶
Encodes an image into base64 encoded string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
img
|
Image
|
The PIL Image object to encode. |
required |
format
|
str
|
The image format to use (e.g., "JPEG", "PNG"). |
required |
Returns:
| Type | Description |
|---|---|
str
|
A base64 encoded string representation of the image. |
Source code in aiperf/services/dataset/utils.py
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | |
load_json_str(json_str, func=lambda x: x)
¶
Deserializes JSON encoded string into Python object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
json_str
|
str
|
JSON encoded string |
required |
func
|
Callable
|
A function that takes deserialized JSON object. This can be used to run validation checks on the object. Defaults to identity function. |
lambda x: x
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The deserialized JSON object. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the JSON string is invalid. |
Source code in aiperf/services/dataset/utils.py
77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
open_image(filename)
¶
Opens an image file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
The file path to open. |
required |
Returns:
| Type | Description |
|---|---|
Image
|
The opened PIL Image object. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If the file does not exist. |
Source code in aiperf/services/dataset/utils.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | |
sample_normal(mean, stddev, lower=-np.inf, upper=np.inf)
¶
Sample from a normal distribution with support for bounds using rejection sampling.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mean
|
float
|
The mean of the normal distribution. |
required |
stddev
|
float
|
The standard deviation of the normal distribution. |
required |
lower
|
float
|
The lower bound of the distribution. |
-inf
|
upper
|
float
|
The upper bound of the distribution. |
inf
|
Returns:
| Type | Description |
|---|---|
int
|
An integer sampled from the distribution. |
Source code in aiperf/services/dataset/utils.py
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | |
sample_positive_normal(mean, stddev)
¶
Sample from a normal distribution ensuring positive values without distorting the distribution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mean
|
float
|
Mean value for the normal distribution |
required |
stddev
|
float
|
Standard deviation for the normal distribution |
required |
Returns:
| Type | Description |
|---|---|
float
|
A positive sample from the normal distribution |
Raises:
| Type | Description |
|---|---|
ValueError
|
If mean is less than 0 |
Source code in aiperf/services/dataset/utils.py
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | |
sample_positive_normal_integer(mean, stddev)
¶
Sample a random positive integer from a normal distribution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mean
|
float
|
The mean of the normal distribution. |
required |
stddev
|
float
|
The standard deviation of the normal distribution. |
required |
Returns:
| Type | Description |
|---|---|
int
|
A positive integer sampled from the distribution. If the sampled |
int
|
number is less than 1, it returns 1. |
Source code in aiperf/services/dataset/utils.py
138 139 140 141 142 143 144 145 146 147 148 149 | |
aiperf.services.inference_result_parser.inference_result_parser¶
InferenceResultParser
¶
Bases: BaseComponentService
InferenceResultParser is responsible for parsing the inference results and pushing them to the RecordsManager.
Source code in aiperf/services/inference_result_parser/inference_result_parser.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 | |
service_type
property
¶
The type of service.
compute_input_token_count(record, tokenizer)
async
¶
Compute the number of tokens in the input for a given request record.
Source code in aiperf/services/inference_result_parser/inference_result_parser.py
217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 | |
get_tokenizer(model)
async
¶
Get the tokenizer for a given model.
Source code in aiperf/services/inference_result_parser/inference_result_parser.py
109 110 111 112 113 114 115 116 117 118 | |
process_valid_record(message)
async
¶
Process a valid request record.
Source code in aiperf/services/inference_result_parser/inference_result_parser.py
182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 | |
aiperf.services.inference_result_parser.openai_parsers¶
OpenAIObject
¶
Bases: CaseInsensitiveStrEnum
Types of OpenAI objects.
Source code in aiperf/services/inference_result_parser/openai_parsers.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
parse(text)
classmethod
¶
Attempt to parse a string into an OpenAI object.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the object is invalid. |
Source code in aiperf/services/inference_result_parser/openai_parsers.py
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
OpenAIResponseExtractor
¶
Extractor for OpenAI responses.
Source code in aiperf/services/inference_result_parser/openai_parsers.py
75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | |
__init__(model_endpoint)
¶
Create a new response extractor based on the provided configuration.
Source code in aiperf/services/inference_result_parser/openai_parsers.py
83 84 85 | |
extract_response_data(record, tokenizer)
async
¶
Extract the text from a server response message.
Source code in aiperf/services/inference_result_parser/openai_parsers.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 | |
aiperf.services.records_manager.metrics.base_metric¶
BaseMetric
¶
Bases: ABC
Base class for all metrics with automatic subclass registration.
Source code in aiperf/services/records_manager/metrics/base_metric.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | |
__init_subclass__(**kwargs)
¶
This method is called when a class is subclassed from Metric.
It automatically registers the subclass in the metric_interfaces
dictionary using the tag class attribute.
The tag attribute must be a non-empty string that uniquely identifies the
metric type. Only concrete (non-abstract) classes will be registered.
Source code in aiperf/services/records_manager/metrics/base_metric.py
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | |
get_all()
classmethod
¶
Returns the dictionary of all registered metric interfaces.
This method dynamically imports all metric type modules from the 'types' directory to ensure all metric classes are registered via init_subclass.
Returns:
| Type | Description |
|---|---|
dict[str, type[BaseMetric]]
|
dict[str, type[Metric]]: Mapping of metric tags to their corresponding classes |
Raises:
| Type | Description |
|---|---|
MetricTypeError
|
If there's an error importing metric type modules |
Source code in aiperf/services/records_manager/metrics/base_metric.py
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | |
update_value(record=None, metrics=None)
abstractmethod
¶
Updates the metric value based on the provided record and dictionary of other metrics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
record
|
Optional[Record]
|
The record to update the metric with. |
None
|
metrics
|
Optional[dict[BaseMetric]]
|
A dictionary of other metrics that may be needed for calculation. |
None
|
Source code in aiperf/services/records_manager/metrics/base_metric.py
88 89 90 91 92 93 94 95 96 97 98 99 100 | |
values()
abstractmethod
¶
Returns the list of calculated metrics.
Source code in aiperf/services/records_manager/metrics/base_metric.py
102 103 104 105 106 | |
aiperf.services.records_manager.metrics.types.benchmark_duration_metric¶
BenchmarkDurationMetric
¶
Bases: BaseMetric
Post-processor for calculating the Benchmark Duration metric.
Source code in aiperf/services/records_manager/metrics/types/benchmark_duration_metric.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | |
values()
¶
Returns the list of Time to First Token (BenchmarkDuration) metrics.
Source code in aiperf/services/records_manager/metrics/types/benchmark_duration_metric.py
38 39 40 41 42 | |
aiperf.services.records_manager.metrics.types.input_sequence_length_metric¶
InputSequenceLengthMetric
¶
Bases: BaseMetric
Post-processor for calculating Input Sequence Length (ISL) metrics from records.
Source code in aiperf/services/records_manager/metrics/types/input_sequence_length_metric.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | |
values()
¶
Returns the list of Input Sequence Length (ISL) metrics.
Source code in aiperf/services/records_manager/metrics/types/input_sequence_length_metric.py
33 34 35 36 37 | |
aiperf.services.records_manager.metrics.types.max_response_metric¶
MaxResponseMetric
¶
Bases: BaseMetric
Post-processor for calculating the maximum response time stamp metric from records.
Source code in aiperf/services/records_manager/metrics/types/max_response_metric.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
update_value(record=None, metrics=None)
¶
Adds a new record and calculates the maximum response timestamp metric.
Source code in aiperf/services/records_manager/metrics/types/max_response_metric.py
22 23 24 25 26 27 28 29 30 31 32 33 | |
values()
¶
Returns the list of Time to First Token (TTFT) metrics.
Source code in aiperf/services/records_manager/metrics/types/max_response_metric.py
35 36 37 38 39 | |
aiperf.services.records_manager.metrics.types.min_request_metric¶
MinRequestMetric
¶
Bases: BaseMetric
Post-processor for calculating the minimum request time stamp metric from records.
Source code in aiperf/services/records_manager/metrics/types/min_request_metric.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
update_value(record=None, metrics=None)
¶
Adds a new record and calculates the minimum request timestamp metric.
Source code in aiperf/services/records_manager/metrics/types/min_request_metric.py
22 23 24 25 26 27 28 29 30 31 32 33 | |
values()
¶
Returns the list of Time to First Token (TTFT) metrics.
Source code in aiperf/services/records_manager/metrics/types/min_request_metric.py
35 36 37 38 39 | |
aiperf.services.records_manager.metrics.types.output_sequence_length_metric¶
OutputSequenceLengthMetric
¶
Bases: BaseMetric
Post-processor for calculating Output Sequence Length (OSL) metrics from records.
Source code in aiperf/services/records_manager/metrics/types/output_sequence_length_metric.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | |
values()
¶
Returns the list of Output Sequence Length (OSL) metrics.
Source code in aiperf/services/records_manager/metrics/types/output_sequence_length_metric.py
33 34 35 36 37 | |
aiperf.services.records_manager.metrics.types.output_token_throughput_per_user_metric¶
OutputTokenThroughputPerUserMetric
¶
Bases: BaseMetric
Post Processor for calculating Output Token Throughput per user metrics from records.
Source code in aiperf/services/records_manager/metrics/types/output_token_throughput_per_user_metric.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | |
values()
¶
Returns the list of Output Token Throughput Per User metrics.
Source code in aiperf/services/records_manager/metrics/types/output_token_throughput_per_user_metric.py
35 36 37 38 39 | |
aiperf.services.records_manager.metrics.types.request_latency_metric¶
RequestLatencyMetric
¶
Bases: BaseMetric
Post-processor for calculating Request Latency metrics from records.
Source code in aiperf/services/records_manager/metrics/types/request_latency_metric.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | |
update_value(record=None, metrics=None)
¶
Adds a new record and calculates the Request Latencies metric.
This method extracts the request and last response timestamps, calculates the differences in time, and appends the result to the metric list.
Source code in aiperf/services/records_manager/metrics/types/request_latency_metric.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | |
values()
¶
Returns the list of Time to First Token (Request Latencies) metrics.
Source code in aiperf/services/records_manager/metrics/types/request_latency_metric.py
39 40 41 42 43 | |
aiperf.services.records_manager.metrics.types.request_throughput_metric¶
RequestThroughputMetric
¶
Bases: BaseMetric
Post Processor for calculating Request throughput metrics from records.
Source code in aiperf/services/records_manager/metrics/types/request_throughput_metric.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | |
values()
¶
Returns the Request Throughput metric.
Source code in aiperf/services/records_manager/metrics/types/request_throughput_metric.py
38 39 40 41 42 | |
aiperf.services.records_manager.metrics.types.ttft_metric¶
TTFTMetric
¶
Bases: BaseMetric
Post-processor for calculating Time to First Token (TTFT) metrics from records.
Source code in aiperf/services/records_manager/metrics/types/ttft_metric.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | |
update_value(record=None, metrics=None)
¶
Adds a new record and calculates the Time To First Token (TTFT) metric.
This method extracts the timestamp from the request and the first response in the given RequestRecord object, computes the difference (TTFT), and appends the result to the metric list.
Source code in aiperf/services/records_manager/metrics/types/ttft_metric.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | |
values()
¶
Returns the list of Time to First Token (TTFT) metrics.
Source code in aiperf/services/records_manager/metrics/types/ttft_metric.py
39 40 41 42 43 | |
aiperf.services.records_manager.metrics.types.ttst_metric¶
TTSTMetric
¶
Bases: BaseMetric
Post-processor for calculating Time to Second Token (TTST) metrics from records.
Source code in aiperf/services/records_manager/metrics/types/ttst_metric.py
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | |
update_value(record=None, metrics=None)
¶
Adds a new record and calculates the Time To Second Token (TTST) metric.
This method extracts the timestamp from the first and second response in the given Record object, computes the difference (TTST), and appends the result to the metric list.
Source code in aiperf/services/records_manager/metrics/types/ttst_metric.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | |
values()
¶
Returns the list of Time to First Token (TTST) metrics.
Source code in aiperf/services/records_manager/metrics/types/ttst_metric.py
39 40 41 42 43 | |
aiperf.services.records_manager.post_processors.metric_summary¶
MetricSummary
¶
MetricSummary is a post-processor that generates a summary of metrics from the records. It processes the records to extract relevant metrics and returns them in a structured format.
Source code in aiperf/services/records_manager/post_processors/metric_summary.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | |
process(records)
¶
Process the records to generate a summary of metrics.
:param records: The input records to be processed. :return: A dictionary containing the summarized metrics.
Source code in aiperf/services/records_manager/post_processors/metric_summary.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | |
record_from_dataframe(df, metric)
¶
Create a Record from a DataFrame.
Source code in aiperf/services/records_manager/post_processors/metric_summary.py
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | |
aiperf.services.records_manager.post_processors.streaming_post_processor¶
BaseStreamingPostProcessor
¶
Bases: AIPerfLifecycleMixin, ABC
BaseStreamingPostProcessor is a base class for all classes that wish to stream the incoming ParsedResponseRecords.
Source code in aiperf/services/records_manager/post_processors/streaming_post_processor.py
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | |
stream_record(record)
abstractmethod
async
¶
Handle the incoming record. This method should be implemented by the subclass.
Source code in aiperf/services/records_manager/post_processors/streaming_post_processor.py
62 63 64 65 66 67 | |
aiperf.services.records_manager.records_manager¶
DEFAULT_MAX_RECORDS_CONCURRENCY = 100000
module-attribute
¶
The default maximum concurrency for the records manager pull client.
RecordsManager
¶
Bases: BaseComponentService
The RecordsManager service is primarily responsible for holding the results returned from the workers.
Source code in aiperf/services/records_manager/records_manager.py
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 | |
service_type
property
¶
The type of service.
get_error_summary()
async
¶
Generate a summary of the error records.
Source code in aiperf/services/records_manager/records_manager.py
253 254 255 256 257 258 259 260 261 262 263 264 265 266 | |
post_process_records()
async
¶
Post process the records.
Source code in aiperf/services/records_manager/records_manager.py
313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 | |
process_records(message)
async
¶
Process the records.
This method is called when the records manager receives a command to process the records.
Source code in aiperf/services/records_manager/records_manager.py
268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 | |
publish_processing_stats()
async
¶
Publish the profile stats.
Source code in aiperf/services/records_manager/records_manager.py
158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 | |
main()
¶
Main entry point for the records manager.
Source code in aiperf/services/records_manager/records_manager.py
350 351 352 353 354 355 | |
aiperf.services.service_manager.base¶
BaseServiceManager
¶
Bases: AIPerfLoggerMixin, ABC
Base class for service managers. It provides a common interface for managing services and a way to look up service information by service ID.
Source code in aiperf/services/service_manager/base.py
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | |
kill_all_services()
abstractmethod
async
¶
Kill all managed services.
Source code in aiperf/services/service_manager/base.py
43 44 45 46 | |
run_all_services()
abstractmethod
async
¶
Run all required services.
Source code in aiperf/services/service_manager/base.py
33 34 35 36 | |
shutdown_all_services()
abstractmethod
async
¶
Shutdown all managed services.
Source code in aiperf/services/service_manager/base.py
38 39 40 41 | |
wait_for_all_services_registration(stop_event, timeout_seconds=30)
abstractmethod
async
¶
Wait for all required services to be registered.
Source code in aiperf/services/service_manager/base.py
48 49 50 51 52 53 | |
aiperf.services.service_manager.kubernetes¶
KubernetesServiceManager
¶
Bases: BaseServiceManager
Service Manager for starting and stopping services in a Kubernetes cluster.
Source code in aiperf/services/service_manager/kubernetes.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | |
kill_all_services()
async
¶
Kill all required services as Kubernetes pods.
Source code in aiperf/services/service_manager/kubernetes.py
50 51 52 53 54 55 56 | |
run_all_services()
async
¶
Initialize all required services as Kubernetes pods.
Source code in aiperf/services/service_manager/kubernetes.py
34 35 36 37 38 39 40 | |
shutdown_all_services()
async
¶
Stop all required services as Kubernetes pods.
Source code in aiperf/services/service_manager/kubernetes.py
42 43 44 45 46 47 48 | |
wait_for_all_services_registration(stop_event, timeout_seconds=30)
async
¶
Wait for all required services to be registered in Kubernetes.
Source code in aiperf/services/service_manager/kubernetes.py
58 59 60 61 62 63 64 65 66 67 68 | |
wait_for_all_services_start()
async
¶
Wait for all required services to be started in Kubernetes.
Source code in aiperf/services/service_manager/kubernetes.py
70 71 72 73 74 75 76 77 78 | |
ServiceKubernetesRunInfo
¶
Bases: BaseModel
Information about a service running in a Kubernetes pod.
Source code in aiperf/services/service_manager/kubernetes.py
13 14 15 16 17 18 | |
aiperf.services.service_manager.multiprocess¶
MultiProcessRunInfo
¶
Bases: BaseModel
Information about a service running as a multiprocessing process.
Source code in aiperf/services/service_manager/multiprocess.py
22 23 24 25 26 27 28 29 30 31 | |
MultiProcessServiceManager
¶
Bases: BaseServiceManager
Service Manager for starting and stopping services as multiprocessing processes.
Source code in aiperf/services/service_manager/multiprocess.py
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 | |
kill_all_services()
async
¶
Kill all required services as multiprocessing processes.
Source code in aiperf/services/service_manager/multiprocess.py
106 107 108 109 110 111 112 113 114 115 116 117 118 | |
run_all_services()
async
¶
Start all required services as multiprocessing processes.
Source code in aiperf/services/service_manager/multiprocess.py
87 88 89 90 91 92 93 94 95 | |
shutdown_all_services()
async
¶
Stop all required services as multiprocessing processes.
Source code in aiperf/services/service_manager/multiprocess.py
97 98 99 100 101 102 103 104 | |
wait_for_all_services_registration(stop_event, timeout_seconds=30)
async
¶
Wait for all required services to be registered.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stop_event
|
Event
|
Event to check if operation should be cancelled |
required |
timeout_seconds
|
int
|
Maximum time to wait in seconds |
30
|
Source code in aiperf/services/service_manager/multiprocess.py
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | |
aiperf.services.system_controller.system_controller¶
SystemController
¶
Bases: SignalHandlerMixin, BaseControllerService
System Controller service.
This service is responsible for managing the lifecycle of all other services. It will start, stop, and configure all other services.
Source code in aiperf/services/system_controller/system_controller.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 | |
service_type
property
¶
The type of service.
initialize()
async
¶
Override the base initialize method to add pre-initialization and post-initialization steps. This allows us to run the UI and progress logger before the system is fully initialized.
Source code in aiperf/services/system_controller/system_controller.py
97 98 99 100 101 102 103 104 | |
kill()
async
¶
Kill the system controller.
Source code in aiperf/services/system_controller/system_controller.py
516 517 518 519 520 521 | |
send_command_to_service(target_service_id, command, data=None, target_service_type=None)
async
¶
Send a command to a specific service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target_service_id
|
str | None
|
ID of the target service, or None to send to all services |
required |
target_service_type
|
ServiceType | None
|
Type of the target service, or None to send to all services |
None
|
command
|
CommandType
|
The command to send (from CommandType enum). |
required |
data
|
Any | None
|
Optional data to send with the command. |
None
|
Raises:
| Type | Description |
|---|---|
CommunicationError
|
If the communication is not initialized or the command was not sent successfully |
Source code in aiperf/services/system_controller/system_controller.py
477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 | |
start_profiling_all_services()
async
¶
Tell all services to start profiling.
Source code in aiperf/services/system_controller/system_controller.py
323 324 325 326 327 328 329 330 331 332 | |
main()
¶
Main entry point for the system controller.
Source code in aiperf/services/system_controller/system_controller.py
524 525 526 527 528 529 | |
aiperf.services.system_controller.system_mixins¶
SignalHandlerMixin
¶
Mixin for services that need to handle system signals.
Source code in aiperf/services/system_controller/system_mixins.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | |
setup_signal_handlers(callback)
¶
This method will set up signal handlers for the SIGTERM and SIGINT signals in order to trigger a graceful shutdown of the service.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
callback
|
Callable[[int], Coroutine[Any, Any, None]]
|
The callback to call when a signal is received |
required |
Source code in aiperf/services/system_controller/system_mixins.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | |
aiperf.services.timing_manager.concurrency_strategy¶
ConcurrencyStrategy
¶
Bases: CreditIssuingStrategy, AsyncTaskManagerMixin, AIPerfLoggerMixin
Class for concurrency credit issuing strategy.
Source code in aiperf/services/timing_manager/concurrency_strategy.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | |
aiperf.services.timing_manager.config¶
TimingManagerConfig
¶
Bases: AIPerfBaseModel
Configuration for the timing manager.
Source code in aiperf/services/timing_manager/config.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | |
from_user_config(user_config)
classmethod
¶
Create a TimingManagerConfig from a UserConfig.
Source code in aiperf/services/timing_manager/config.py
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | |
aiperf.services.timing_manager.credit_issuing_strategy¶
CreditIssuingStrategy
¶
Bases: AsyncTaskManagerMixin, AIPerfLoggerMixin, ABC
Base class for credit issuing strategies.
Source code in aiperf/services/timing_manager/credit_issuing_strategy.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 | |
start()
async
¶
Start the credit issuing strategy. This will launch the progress reporting loop, the warmup phase (if applicable), and the profiling phase, all in the background.
Source code in aiperf/services/timing_manager/credit_issuing_strategy.py
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | |
stop()
async
¶
Stop the credit issuing strategy.
Source code in aiperf/services/timing_manager/credit_issuing_strategy.py
134 135 136 | |
CreditIssuingStrategyFactory
¶
Bases: FactoryMixin[TimingMode, CreditIssuingStrategy]
Factory for creating credit issuing strategies based on the timing mode.
Source code in aiperf/services/timing_manager/credit_issuing_strategy.py
194 195 | |
aiperf.services.timing_manager.credit_manager¶
CreditManagerProtocol
¶
Bases: Protocol
Defines the interface for a CreditManager.
This is used to allow the credit issuing strategy to interact with the TimingManager in a decoupled way.
Source code in aiperf/services/timing_manager/credit_manager.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | |
CreditPhaseMessagesMixin
¶
Bases: AsyncTaskManagerMixin, CreditPhaseMessagesRequirements
Mixin for services to implement the CreditManagerProtocol.
Requirements
This mixin must be used with a class that provides: - pub_client: PubClientProtocol - service_id: str
Source code in aiperf/services/timing_manager/credit_manager.py
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 | |
publish_credits_complete()
async
¶
Publish the credits complete message.
Source code in aiperf/services/timing_manager/credit_manager.py
152 153 154 155 156 157 | |
publish_phase_complete(phase, completed, end_ns)
async
¶
Publish the phase complete message.
Source code in aiperf/services/timing_manager/credit_manager.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 | |
publish_phase_sending_complete(phase, sent_end_ns)
async
¶
Publish the phase sending complete message.
Source code in aiperf/services/timing_manager/credit_manager.py
108 109 110 111 112 113 114 115 116 117 118 119 120 | |
publish_phase_start(phase, start_ns, total_expected_requests, expected_duration_sec)
async
¶
Publish the phase start message.
Source code in aiperf/services/timing_manager/credit_manager.py
87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | |
publish_progress(phase, sent, completed)
async
¶
Publish the progress message.
Source code in aiperf/services/timing_manager/credit_manager.py
137 138 139 140 141 142 143 144 145 146 147 148 149 150 | |
CreditPhaseMessagesRequirements
¶
Bases: AsyncTaskManagerProtocol, AIPerfLoggerProtocol, Protocol
Requirements for the CreditPhaseMessagesMixin. This is the list of attributes that must be provided by the class that uses this mixin.
Source code in aiperf/services/timing_manager/credit_manager.py
60 61 62 63 64 65 66 67 68 | |
aiperf.services.timing_manager.fixed_schedule_strategy¶
FixedScheduleStrategy
¶
Bases: CreditIssuingStrategy, AsyncTaskManagerMixin
Class for fixed schedule credit issuing strategy.
Source code in aiperf/services/timing_manager/fixed_schedule_strategy.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | |
aiperf.services.timing_manager.request_rate_strategy¶
RequestRateStrategy
¶
Bases: CreditIssuingStrategy, AsyncTaskManagerMixin
Strategy for issuing credits based on a specified request rate.
Supports two modes: - CONSTANT: Issues credits at a constant rate with fixed intervals - POISSON: Issues credits using a Poisson process with exponentially distributed intervals
Source code in aiperf/services/timing_manager/request_rate_strategy.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | |
aiperf.services.timing_manager.timing_manager¶
TimingManager
¶
Bases: BaseComponentService, CreditPhaseMessagesMixin
The TimingManager service is responsible to generate the schedule and issuing timing credits for requests.
Source code in aiperf/services/timing_manager/timing_manager.py
47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 | |
service_type
property
¶
The type of service.
drop_credit(credit_phase, conversation_id=None, credit_drop_ns=None)
async
¶
Drop a credit.
Source code in aiperf/services/timing_manager/timing_manager.py
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 | |
main()
¶
Main entry point for the timing manager.
Source code in aiperf/services/timing_manager/timing_manager.py
194 195 196 197 198 | |
aiperf.services.workers.credit_processor_mixin¶
CreditProcessorMixin
¶
Bases: CreditProcessorMixinRequirements
CreditProcessorMixin is a mixin that provides a method to process credit drops.
Source code in aiperf/services/workers/credit_processor_mixin.py
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 | |
CreditProcessorMixinRequirements
¶
Bases: AIPerfLoggerProtocol, Protocol
CreditProcessorMixinRequirements is a protocol that provides the requirements needed for the CreditProcessorMixin.
Source code in aiperf/services/workers/credit_processor_mixin.py
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | |
CreditProcessorProtocol
¶
Bases: Protocol
CreditProcessorProtocol is a protocol that provides a method to process credit drops.
Source code in aiperf/services/workers/credit_processor_mixin.py
29 30 31 32 33 34 35 36 37 | |
aiperf.services.workers.worker¶
Worker
¶
Bases: BaseComponentService, ProcessHealthMixin, CreditProcessorMixin
Worker is primarily responsible for making API calls to the inference server. It also manages the conversation between turns and returns the results to the Inference Results Parsers.
Source code in aiperf/services/workers/worker.py
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 | |
aiperf.services.workers.worker_manager¶
WorkerManager
¶
Bases: BaseComponentService
The WorkerManager service is primary responsibility to manage the worker processes. It will spawn the workers, monitor their health, and stop them when the service is stopped. In the future it will also be responsible for the auto-scaling of the workers.
Source code in aiperf/services/workers/worker_manager.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 | |
WorkerProcessInfo
¶
Bases: AIPerfBaseModel
Information about a worker process.
Source code in aiperf/services/workers/worker_manager.py
29 30 31 32 33 34 35 | |